Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add checkpoint writer #280

Merged
merged 22 commits into from
Jun 11, 2021
Merged

Add checkpoint writer #280

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
2f7d3c8
Add checkpoint writer (wip)
xianwill Jun 2, 2021
c7caaec
Remove unnecessary code line
xianwill Jun 2, 2021
5c0186d
fix checkpoint file name
xianwill Jun 2, 2021
a5c05e9
Add DeltaLogSchemaFactory
xianwill Jun 5, 2021
1d7ea53
merge main
xianwill Jun 5, 2021
794bab3
Adding tests
xianwill Jun 5, 2021
34d8c07
Fix clippy errors
xianwill Jun 5, 2021
8fff84c
Remove unnecessary error sources
xianwill Jun 5, 2021
ec9c5ff
Add checkpoint_for_version method
xianwill Jun 6, 2021
503cd61
Fix debug statement
xianwill Jun 8, 2021
ea1c442
avoid unnecessary clones
Jun 8, 2021
dca5728
Merge pull request #1 from delta-io/qp_clone
xianwill Jun 9, 2021
0f7dba5
Remove redundant clones
xianwill Jun 9, 2021
3bfb2ca
Move CheckPointWriter to checkpoints mod
xianwill Jun 9, 2021
5d5d26d
Stop calling URIs as paths in checkpoint writer
xianwill Jun 9, 2021
5451274
Change checkpoint writer action serialization to use decoder
xianwill Jun 9, 2021
ef16674
Write Arrow schema for delta log directly instead of deserializing fr…
xianwill Jun 10, 2021
68217dd
Fix clippy warnings
xianwill Jun 10, 2021
44340e5
Add unit test for delta log schema and issue links for checkpoint enh…
xianwill Jun 10, 2021
7f878a8
Remove commented code from schema.rs
xianwill Jun 10, 2021
0b22619
Remove unnecessary collect when building checkpoint
xianwill Jun 11, 2021
e6a0fc4
Remove useless into_iter
xianwill Jun 11, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 41 additions & 20 deletions Cargo.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 6 additions & 4 deletions rust/src/action.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -612,7 +612,7 @@ pub struct Txn {
/// An application-specific numeric identifier for this transaction.
pub version: DeltaDataTypeVersion,
/// The time when this transaction action was created in milliseconds since the Unix epoch.
pub last_updated: DeltaDataTypeTimestamp,
pub last_updated: Option<DeltaDataTypeTimestamp>,
}

impl Txn {
Expand All @@ -635,9 +635,11 @@ impl Txn {
.map_err(|_| gen_action_type_error("txn", "version", "long"))?;
}
"lastUpdated" => {
re.last_updated = record
.get_long(i)
.map_err(|_| gen_action_type_error("txn", "lastUpdated", "long"))?;
re.last_updated = Some(
record
.get_long(i)
.map_err(|_| gen_action_type_error("txn", "lastUpdated", "long"))?,
);
}
_ => {
log::warn!(
Expand Down
Loading