Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: fast XML upload (DEV-1626) #352

Merged
merged 220 commits into from May 15, 2023
Merged

feat: fast XML upload (DEV-1626) #352

merged 220 commits into from May 15, 2023

Conversation

irinaschubert
Copy link

resolves DEV-1626

@jnussbaum
working version
75d8c01
@jnussbaum
begin with docs
4aa334b
@jnussbaum
Merge branch 'main' into wip/dev-1626-new-command-mass-upload
69c8120
@jnussbaum
add CLI interface
aa87902
@jnussbaum
generate test data is finished
640027d
@jnussbaum
fix mkdocs.yml
fad78a5
@jnussbaum
refactor
73cd3e5
@jnussbaum
files can be uploaded,
c002c61 
mapping of original filepath to internal filename is created
@jnussbaum
edit
9c23be4
@jnussbaum
edit
17b18ef
@jnussbaum
add ZIP packaging
3dcc7bd
@jnussbaum
fix
8db55aa
@jnussbaum
fix (2)
3d79a4c
@jnussbaum
make file packaging thread-safe
1a0f26f
@jnussbaum
test data generation: generate XML file
21849c4
@jnussbaum
implement check_multimedia_folder()
dbde961
@jnussbaum
add TODO
c8657a9
@jnussbaum
bugfix
3feb28d 
assert that ZIP size is consistent with original batch size
@jnussbaum
remove outdated comments
14236d9
@jnussbaum
edit
1f6ab99
@jnussbaum
distribute the paths into batchgroups
cd8245c
@jnussbaum
improve typing and variable names
3b59186
@jnussbaum
better console output
2548828
@jnussbaum
improve docstrings and user output
570f548
@jnussbaum
bugfix: batch IDs must contain group ID, otherwise they will be overw…
b5693d7 
…ritten
@jnussbaum
bugfix: check_multimedia_folder() didn't work
31bca78
@jnussbaum
refactor check_multimedia_folder()
054107c
@jnussbaum
determine size of batchgroup by number of threads
50d8fbb
@jnussbaum
experimental: make_equally_sized_batches()
2ba4082
@jnussbaum
rename enhanced-xmlupload to preprocess-xmlupload
8676b54
irinaschubert
irinaschubert commented May 11, 2023
View reviewed changes
docs/internal/fast-xmlupload.md Outdated Show resolved Hide resolved
docs/internal/fast-xmlupload.md Outdated Show resolved Hide resolved
src/dsp_tools/fast_xmlupload/process_files.py Outdated Show resolved Hide resolved
src/dsp_tools/fast_xmlupload/process_files.py Show resolved Hide resolved
src/dsp_tools/fast_xmlupload/process_files.py Show resolved Hide resolved
src/dsp_tools/fast_xmlupload/process_files.py Outdated Show resolved Hide resolved
src/dsp_tools/fast_xmlupload/upload_files.py Outdated Show resolved Hide resolved
src/dsp_tools/fast_xmlupload/upload_files.py Outdated Show resolved Hide resolved
src/dsp_tools/fast_xmlupload/upload_files.py Outdated Show resolved Hide resolved
src/dsp_tools/fast_xmlupload/upload_files.py Outdated Show resolved Hide resolved
@irinaschubert irinaschubert removed their assignment May 11, 2023
@jnussbaum jnussbaum merged commit c2f46ea into main May 15, 2023
9 checks passed
@jnussbaum jnussbaum deleted the wip/fast-xml-upload branch May 15, 2023 07:28
@daschbot daschbot mentioned this pull request May 15, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BalduinLandolt BalduinLandolt BalduinLandolt left review comments

@jnussbaum jnussbaum jnussbaum approved these changes

Assignees

@jnussbaum jnussbaum

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@irinaschubert @BalduinLandolt @jnussbaum @subotic