Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

oss: migrate to ossfs #6307

Merged
merged 23 commits into from
Jul 13, 2021
Merged

oss: migrate to ossfs #6307

merged 23 commits into from
Jul 13, 2021

Commits on Jul 13, 2021

  1. Migrate to ossfs 

    fix 6177
    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    e67fed4 View commit details
    Browse the repository at this point in the history

  2. Add verification to object stored on oss

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    57a4493 View commit details
    Browse the repository at this point in the history

  3. Change the oss remote's dependency. 

    pass all of the tests
    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    f44127e View commit details
    Browse the repository at this point in the history

  4. Performance optimaztion and ossfs version update

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    52f3206 View commit details
    Browse the repository at this point in the history

  5. Remove the argument to set the resumable get/put 

    1. Experiments shows resumable get/put has little influnence on performance.
2. Just use resumable ones as default
    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    3940f0a View commit details
    Browse the repository at this point in the history

  6. Use a new api for performance optimization

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    6420e1e View commit details
    Browse the repository at this point in the history

  7. Without filesize progress bar version

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    f160f38 View commit details
    Browse the repository at this point in the history

  8. With progress bar version

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    5fa18a6 View commit details
    Browse the repository at this point in the history

  9. Bump and add comments 

    1. Bump ossfs from 2021.6.2 to 2021.7.0
2. add some comments to the verification trigger
    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    da67a8f View commit details
    Browse the repository at this point in the history

  10. Change the name of kwrags following parent class.

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    9f3c1f2 View commit details
    Browse the repository at this point in the history

  11. Bump to new ossfs and rewrite upload_fobj 

    Because some problem with oss-emulater
1. bump to ossfs 2021.7.1
2. rewrite upload_fobj
    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    f3b972c View commit details
    Browse the repository at this point in the history

  12. Optimization on rm_files

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    64e8563 View commit details
    Browse the repository at this point in the history

  13. Some optimization follow the old version.

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    5d1bb37 View commit details
    Browse the repository at this point in the history

  14. Update dvc/fs/oss.py 

    Co-authored-by: Ruslan Kuprieiev <kupruser@gmail.com>
    @karajan1001 @efiop
    karajan1001 and efiop committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    9a9c0ee View commit details
    Browse the repository at this point in the history

  15. Update dvc/objects/db/oss.py 

    Co-authored-by: Ruslan Kuprieiev <kupruser@gmail.com>
    @karajan1001 @efiop
    karajan1001 and efiop committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    3524c62 View commit details
    Browse the repository at this point in the history

  16. bump into ossfs 2021.7.2

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    68aae40 View commit details
    Browse the repository at this point in the history

  17. Bump into ossfs 2021.7.3OC

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    546a715 View commit details
    Browse the repository at this point in the history

  18. Switch to real_oss_server

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    10e6a30 View commit details
    Browse the repository at this point in the history

  19. Switch to a real oss server

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    3284b2f View commit details
    Browse the repository at this point in the history

  20. Use the oss bucket located in US.

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    1bada76 View commit details
    Browse the repository at this point in the history

  21. Remove some emulator related code

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    efaa864 View commit details
    Browse the repository at this point in the history

  22. Add envs to ci

    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    c1b4941 View commit details
    Browse the repository at this point in the history

  23. remove some oss pytest XFAIL 

    Because we didn't use emulator, we can remove some pytest XFAILs
    @karajan1001
    karajan1001 committed Jul 13, 2021
    Configuration menu
    Copy the full SHA
    b238518 View commit details
    Browse the repository at this point in the history