Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: resume an interrupted xmlupload (DEV-3323) #853

Merged
merged 35 commits into from Mar 7, 2024
Merged

feat: resume an interrupted xmlupload (DEV-3323) #853

merged 35 commits into from Mar 7, 2024

Conversation

jnussbaum
Copy link
Collaborator

No description provided.

@jnussbaum jnussbaum self-assigned this Mar 4, 2024
@linear Linear
Copy link

linear bot commented Mar 4, 2024

DEV-3323 Add the possibility to resume an interrupted xmlupload

jnussbaum and others added 24 commits March 4, 2024 15:32
@jnussbaum
Merge branch 'main' into wip/DEV-3323-resume-an-interrupted-xmlupload
7632753
@jnussbaum
Merge branch 'main' into wip/DEV-3323-resume-an-interrupted-xmlupload
4913943
@jnussbaum
Merge branch 'main' into wip/DEV-3323-resume-an-interrupted-xmlupload
9964005
@jnussbaum
continue
fb476e6
@jnussbaum
tests: write pickle into tmpdir instead of ~/.dsp-tools
135322d
@jnussbaum
refactor
a726a73
@jnussbaum
add permissions_lookup to UploadState
a62570b
@jnussbaum
refactor
5fbacaa
@jnussbaum
add input_file to UploadConfig
aa2e37e
@jnussbaum
add resume-xmlupload to CLI
65dcc2c
@jnussbaum
fix argparse
f4fe209
@jnussbaum
add docstring
fa85a0f
@jnussbaum
keep pending_stash and nonapplied_stash separate; nonapplied_stash mu…
271ae45 
…st not lead to _handle_upload_error()
@jnussbaum
Merge branch 'main' into wip/DEV-3323-resume-an-interrupted-xmlupload
1bb7b04
@jnussbaum
rename stash to pending_stash
4160f14
@jnussbaum
fix
ce4f085
@jnussbaum
remove timestamp from DiagnosticsConfig
473b287
@jnussbaum
fix e2e
f37ecb7
@jnussbaum
typo
8adca28
@jnussbaum
edit
edb1f93
@jnussbaum
fix parser
b7ec4d3
@jnussbaum
fix counter
71647dd
@jnussbaum
inject iri_resolver
61771e9
@BalduinLandolt
add documentation
de447d6
Nora-Olivia-Ammann
Nora-Olivia-Ammann reviewed Mar 7, 2024
View reviewed changes
Copy link
Collaborator

@Nora-Olivia-Ammann Nora-Olivia-Ammann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the file: src/dsp_tools/commands/xmlupload/resource_create_client.py
line: 48.
What happens if the post is already done but we do not have the answer from the server yet but it is a sucessful upload?

If there is no interrupted upload to finish i get a non-200 error, which it retries... that is for a future PR but we should keep in mind to clean that up.

Bad news: The number of triples in the db is not the same after an interrupted upload than a normal one:

  • Regular xmlupload project systematic: 3735
  • Test 1: Interrupted 2 times: 3763
  • Test 2: interrupted 1 times: 3766
    Both interruptions were not during the stash upload.
    I have not done more precise investigations what happened, but clearly something is wrong.

docs/cli-commands.md Outdated Show resolved Hide resolved
src/dsp_tools/commands/xmlupload/stash/upload_stashed_resptr_props.py Show resolved Hide resolved
@jnussbaum
Copy link
Collaborator Author

In the file: src/dsp_tools/commands/xmlupload/resource_create_client.py line: 48. What happens if the post is already done but we do not have the answer from the server yet but it is a sucessful upload?

If there is no interrupted upload to finish i get a non-200 error, which it retries... that is for a future PR but we should keep in mind to clean that up.

Bad news: The number of triples in the db is not the same after an interrupted upload than a normal one:

* Regular xmlupload project systematic: 3735

* Test 1: Interrupted 2 times: 3763

* Test 2: interrupted 1 times: 3766
  Both interruptions were not during the stash upload.
  I have not done more precise investigations what happened, but clearly something is wrong.

Thanks a lot for your valuable feedback. We noted these points and created a follow-up ticket: https://linear.app/dasch/issue/DEV-3392/tidy-up-resume-xmlupload-follow-up

@jnussbaum jnussbaum enabled auto-merge (squash) March 7, 2024 16:14
@jnussbaum jnussbaum merged commit 4acf841 into main Mar 7, 2024
10 checks passed
@jnussbaum jnussbaum deleted the wip/DEV-3323-resume-an-interrupted-xmlupload branch March 7, 2024 16:14
@daschbot daschbot mentioned this pull request Mar 7, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Nora-Olivia-Ammann Nora-Olivia-Ammann Nora-Olivia-Ammann left review comments

@BalduinLandolt BalduinLandolt BalduinLandolt approved these changes

Assignees

@jnussbaum jnussbaum

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@jnussbaum @BalduinLandolt @Nora-Olivia-Ammann