Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use tusd datastorages #4148

Closed
wants to merge 81 commits into from
Closed

Commits on Nov 1, 2023

  1. use tusd datastorages

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    filter metadata
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    calculate sizediff on demand
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    fix empty bucketid
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    add changelog
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    drop unnecessary flag from cleanup
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    fix tests
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    drop unused variable
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    fix unit tests
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    move Finalize to upload package
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    move more functions to upload package
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    propagate sizediff on BytesReady and Revert on failure
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    reuse tus datastore from fs
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    always set file length when initiating uploads
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    log error if we cannot terminate upload
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    make linter happy
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    always send upload length
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    use s3ng config for datastore as well
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    allow configuring upload object prefixes and temp dir
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    fix s3ng test config
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    fix chunking v1
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    fix chunking v1 --sign
    
    open existing file
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    restore size diff calculation
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    treat 0 byte uploads as size deferred
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    return empty reader for 0 byte files
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    comment change that always sends filesize in upload opaque
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    remove commented code
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    fix legacy chunking on s3
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    comment size to check if that causes nc tests te fail
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    run single failing test
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    drop unused option
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    
    ignore mlock files when iterating revisions
    
    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    1293660 View commit details
    Browse the repository at this point in the history
  2. add comments

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    7d2a741 View commit details
    Browse the repository at this point in the history
  3. test msgpack metadata

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    27936aa View commit details
    Browse the repository at this point in the history
  4. run all tests again

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    11ebb76 View commit details
    Browse the repository at this point in the history
  5. test copy

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    80cad58 View commit details
    Browse the repository at this point in the history
  6. log error when trying to read upload info

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    21301b4 View commit details
    Browse the repository at this point in the history
  7. log upload info details when copying fails

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    6d1eb80 View commit details
    Browse the repository at this point in the history
  8. log as error

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 1, 2023
    Configuration menu
    Copy the full SHA
    aa14928 View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2023

  1. reinitialze logger

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    eaf5632 View commit details
    Browse the repository at this point in the history
  2. more logging

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    69b14f5 View commit details
    Browse the repository at this point in the history
  3. declare length

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 2, 2023
    Configuration menu
    Copy the full SHA
    fb88329 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2023

  1. use oc mtime as revision mtime

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    32f5141 View commit details
    Browse the repository at this point in the history
  2. fix lint

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    1afb42a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1aaec6c View commit details
    Browse the repository at this point in the history
  4. declare length only if it is available

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    96ece53 View commit details
    Browse the repository at this point in the history

Commits on Nov 13, 2023

  1. allow overwriting existing revisions

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 13, 2023
    Configuration menu
    Copy the full SHA
    b421107 View commit details
    Browse the repository at this point in the history
  2. do not hide current revision

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 13, 2023
    Configuration menu
    Copy the full SHA
    366dbd4 View commit details
    Browse the repository at this point in the history
  3. Revert "do not hide current revision"

    This reverts commit 366dbd4.
    butonic committed Nov 13, 2023
    Configuration menu
    Copy the full SHA
    303440a View commit details
    Browse the repository at this point in the history
  4. persist revisions by current time, not oc mtime

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 13, 2023
    Configuration menu
    Copy the full SHA
    8fc873a View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2023

  1. filter multiple revisions for the same mtime in ocdav

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    30ab0b6 View commit details
    Browse the repository at this point in the history
  2. use metadata to store and filter current revision

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    2204836 View commit details
    Browse the repository at this point in the history
  3. use oc mtime as revision time if given

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    7e295be View commit details
    Browse the repository at this point in the history
  4. reuse current revision if upload has same mtime

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    1b8c07c View commit details
    Browse the repository at this point in the history
  5. use mtime as revision time if given

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    052b04a View commit details
    Browse the repository at this point in the history

Commits on Nov 15, 2023

  1. track uploads in dedicated metadata

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    7bf1c79 View commit details
    Browse the repository at this point in the history
  2. convert uploads to revisions

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    0b7abc5 View commit details
    Browse the repository at this point in the history
  3. restore old revision logic

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 15, 2023
    Configuration menu
    Copy the full SHA
    0df306e View commit details
    Browse the repository at this point in the history

Commits on Nov 16, 2023

  1. fix blobid passing

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    d75bf9e View commit details
    Browse the repository at this point in the history
  2. fix HasUploadMetadata interface

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    0077f19 View commit details
    Browse the repository at this point in the history
  3. fix legacy chunking

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    a832581 View commit details
    Browse the repository at this point in the history
  4. fix chinkung v1

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    84b9b8f View commit details
    Browse the repository at this point in the history
  5. only create revision on first overwrite

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    1ec133d View commit details
    Browse the repository at this point in the history
  6. restore writing revision before async processing

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    c520b49 View commit details
    Browse the repository at this point in the history

Commits on Nov 17, 2023

  1. fix oldchunking revisions

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    ba50382 View commit details
    Browse the repository at this point in the history
  2. refactor revisionkey split and join

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    90092e4 View commit details
    Browse the repository at this point in the history
  3. no deet to store the current revision, use mtime

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    01b80cd View commit details
    Browse the repository at this point in the history
  4. fix mtime handling

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    508780c View commit details
    Browse the repository at this point in the history
  5. try to fix versions

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 17, 2023
    Configuration menu
    Copy the full SHA
    53c509b View commit details
    Browse the repository at this point in the history

Commits on Nov 20, 2023

  1. Configuration menu
    Copy the full SHA
    b91e7b3 View commit details
    Browse the repository at this point in the history
  2. make test helper always send size

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    a4b88d2 View commit details
    Browse the repository at this point in the history
  3. fix nc upload mock

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    b4cbd8e View commit details
    Browse the repository at this point in the history
  4. more test fixes

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 20, 2023
    Configuration menu
    Copy the full SHA
    ea4d951 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2023

  1. revert dying if dataprovider does not support tus

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    d790238 View commit details
    Browse the repository at this point in the history
  2. tus datatx logging cleanup

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    26524bf View commit details
    Browse the repository at this point in the history
  3. remove unused enable_home config for decomposedfs

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    88d4cb9 View commit details
    Browse the repository at this point in the history
  4. cleanup

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    aae59e3 View commit details
    Browse the repository at this point in the history
  5. fix linting

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    ad21320 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    39c6bb4 View commit details
    Browse the repository at this point in the history
  7. always send upload length

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    12b9979 View commit details
    Browse the repository at this point in the history
  8. if index does not exist return empty list of matches

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 22, 2023
    Configuration menu
    Copy the full SHA
    7bfc0e7 View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2023

  1. do not require sending upload length

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 23, 2023
    Configuration menu
    Copy the full SHA
    22f68c1 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2023

  1. Configuration menu
    Copy the full SHA
    25ad42b View commit details
    Browse the repository at this point in the history
  2. also send content length to dataprovider

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    b9e0f3e View commit details
    Browse the repository at this point in the history
  3. fmt.Println debugging

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Nov 24, 2023
    Configuration menu
    Copy the full SHA
    f3b262b View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2023

  1. Configuration menu
    Copy the full SHA
    98250c7 View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2023

  1. More logging

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    aduffeck authored and butonic committed Nov 28, 2023
    Configuration menu
    Copy the full SHA
    e8155c8 View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2023

  1. Configuration menu
    Copy the full SHA
    3538d62 View commit details
    Browse the repository at this point in the history
  2. More logging

    aduffeck committed Nov 29, 2023
    Configuration menu
    Copy the full SHA
    cd73b5c View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2023

  1. Configuration menu
    Copy the full SHA
    767df60 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5e8383c View commit details
    Browse the repository at this point in the history
  3. remove debug logging

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    89f9606 View commit details
    Browse the repository at this point in the history
  4. more cleanup

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Dec 4, 2023
    Configuration menu
    Copy the full SHA
    a5edbe5 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2023

  1. Apply suggestions from code review

    Co-authored-by: Andre Duffeck <aduffeck@users.noreply.github.com>
    butonic and aduffeck authored Dec 5, 2023
    Configuration menu
    Copy the full SHA
    d99a424 View commit details
    Browse the repository at this point in the history
  2. use log from context

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    2d95dcc View commit details
    Browse the repository at this point in the history
  3. simplify if conditions

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    87916ac View commit details
    Browse the repository at this point in the history
  4. fix typo

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    5053013 View commit details
    Browse the repository at this point in the history
  5. fix listing and purging upload sessions

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    1a35958 View commit details
    Browse the repository at this point in the history
  6. drop comment

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    ea559e1 View commit details
    Browse the repository at this point in the history
  7. fix upload progress creation and read offset from tus upload

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    829df6d View commit details
    Browse the repository at this point in the history
  8. read uploads concurrently

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    747f8f7 View commit details
    Browse the repository at this point in the history
  9. fix loading upload sessions

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Dec 5, 2023
    Configuration menu
    Copy the full SHA
    b643432 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    5c9beca View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    9c2ed70 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2023

  1. Configuration menu
    Copy the full SHA
    9167e4b View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2023

  1. Configuration menu
    Copy the full SHA
    086e321 View commit details
    Browse the repository at this point in the history
  2. Clarify naming

    aduffeck committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    c11ca05 View commit details
    Browse the repository at this point in the history
  3. WIP: Get rid of separate tus fileinfo metadata

    Everything will be stored in the upload session which we already have
    for our own metadata anyway.
    aduffeck committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    be393b3 View commit details
    Browse the repository at this point in the history
  4. make filestore upload session compatible

    Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
    butonic committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    f8264b5 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    7d3ae90 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    d41ec1e View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    e3ba91d View commit details
    Browse the repository at this point in the history