remote: add support for aliyun oss #1961

nanaya-tachibana · 2019-05-02T07:44:19Z

#1954

Have you followed the guidelines in our
Contributing document?
Does your PR affect documented changes or does it add new functionality
that should be documented? If yes, have you created a PR for
dvc.org documenting it or at
least opened an issue for it? If so, please add a link to it. link

efiop · 2019-05-02T09:59:23Z

Hi @nanaya-tachibana !

Thanks for submitting your PR! Looks pretty good! 🙂

Please see that there are some failing tests;
Is there something we could use to test oss without an oss account? Something like azurite for azure maybe? And maybe there is something we could use for unit tests too?

Thanks,
Ruslan

dvc/cache.py

dvc/config.py

dvc/repo/__init__.py

dvc/repo/gc.py

See treeverse#1961.

nanaya-tachibana · 2019-05-04T13:11:34Z

Hi @nanaya-tachibana !

Thanks for submitting your PR! Looks pretty good! 🙂

Please see that there are some failing tests;

Is there something we could use to test oss without an oss account? Something like azurite for azure maybe? And maybe there is something we could use for unit tests too?

Thanks,
Ruslan

I found an old oss emulate and made it compatible with the new oss python SDK.
Now we can test oss by running an emulate in docker.

Start a container running an oss emulator.
$ git clone https://github.com/nanaya-tachibana/oss-emulator.git
$ docker image build -t oss:1.0 oss-emulator
$ docker run --detach -p 8880:8880 --name oss-emulator oss:1.0
Setup environment variables.
$ export OSS_BUCKET='my-bucket'
$ export OSS_ENDPOINT='localhost:8880'
$ export OSS_ACCESS_KEY_ID='AccessKeyID'
$ export OSS_ACCESS_KEY_SECRET='AccessKeySecret'

I also added these processes to travis and appveyor setting files. It was my first time using these continuous integration services. I hope I did correctly.

efiop

Looks great! Please see a few more comments down below.

efiop · 2019-05-06T09:59:46Z

tests/func/test_data_cloud.py

Is this needed? 🙂

Is it safe to remove cli test? I commented this test because the gc cli test failed after I removed the codes in cache.py.

GC cli test? Google Cloud? Or garbage collector?

https://github.com/iterative/dvc/blob/6da0dd6f1cf78900164cee3e4f48763be4104efd/tests/func/test_data_cloud.py#L512-L517

Test for dvc gc command.

And what was the error?

Traceback (most recent call last):
File "/Users/nanaya/Projects/dvc/dvc/main.py", line 38, in main
ret = cmd.run_cmd()
File "/Users/nanaya/Projects/dvc/dvc/command/base.py", line 60, in run_cmd
return self.run()
File "/Users/nanaya/Projects/dvc/dvc/command/gc.py", line 47, in run
repos=self.args.repos,
File "/Users/nanaya/Projects/dvc/dvc/repo/gc.py", line 113, in gc
_do_gc("remote", self.cloud._get_cloud(remote, "gc -c").gc, clist)
File "/Users/nanaya/Projects/dvc/dvc/repo/gc.py", line 54, in _do_gc
removed = func(clist)
File "/Users/nanaya/Projects/dvc/dvc/remote/base.py", line 489, in gc
used |= {info[self.PARAM_CHECKSUM] for info in cinfos[self.scheme]}
KeyError: 'oss'

cinfos is a dict which only has keys for 'local', 's3', 'gs', 'hdfs', 'ssh' and 'azure'.

@nanaya-tachibana Thanks! Looks like a bug. Does replacing cinfos[self.scheme] with cinfos.get(self.scheme, []) help?

@nanaya-tachibana Thanks! Looks like a bug. Does replacing cinfos[self.scheme] with cinfos.get(self.scheme, []) help?

That can fix it. Should I add another commit to fix it and keep the command line tests?

@nanaya-tachibana If you would be so kind 🙂

Ok, decided to help out a little bit 🙂

dvc/remote/oss.py

efiop · 2019-05-07T11:22:31Z

@nanaya-tachibana Btw, please rebase your patch on top of master.

efiop · 2019-05-07T15:44:18Z

@nanaya-tachibana looks like you didn't rebase correctly. Try something like:

git remote add upstream https://github.com/iterative/dvc
git fetch upstream
git rebase upstream/master
git push -f

Usage: $ dvc remote add myremote oss://my-bucket.endpoint/path Set key id and key secret using modify command $ dvc remote modify myremote oss_key_id my-key-id $ dvc remote modify myremote oss_key_secret my-key-secret or environment variables $ export OSS_ACCESS_KEY_ID="my-key-id" $ export OSS_ACCESS_KEY_SECRET="my-key-secret" Ref: oss python SDK: https://www.alibabacloud.com/help/doc-detail/32026.htm

See treeverse#1961.

Start a container running an oss emulator. $ git clone https://github.com/nanaya-tachibana/oss-emulator.git $ docker image build -t oss:1.0 oss-emulator $ docker run --detach -p 8880:8880 --name oss-emulator oss:1.0 Setup environment variables. $ export OSS_BUCKET='my-bucket' $ export OSS_ENDPOINT='localhost:8880' $ export OSS_ACCESS_KEY_ID='AccessKeyID' $ export OSS_ACCESS_KEY_SECRET='AccessKeySecret'

which gives read access to public read bucket and public bucket.

…able value. Usage: $ dvc remote add myremote oss://my-bucket/path Set key id, key secret and endpoint using modify command $ dvc remote modify myremote oss_key_id my-key-id $ dvc remote modify myremote oss_key_secret my-key-secret $ dvc remote modify myremote oss_endpoint endpoint or environment variables $ export OSS_ACCESS_KEY_ID="my-key-id" $ export OSS_ACCESS_KEY_SECRET="my-key-secret" $ export OSS_ENDPOINT="endpoint"

Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai>

Just to keep things in-house. Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai>

efiop · 2019-05-09T12:35:28Z

@nanaya-tachibana Truly outstanding work! 🥇 It is a pleasure working with you! Thank you so much! 🙂

ghost · 2019-05-10T21:00:32Z

dvc/remote/oss.py

+
+class RemoteOSS(RemoteBase):
+    """
+    oss2 document:


efiop · 2019-05-10T21:30:43Z

Hi @nanaya-tachibana !

Thanks again for your contribution! We would love to have a chance to meet you :) Would you be willing to have a hangouts call with us some time? If so, please ping me through email(in my profile) or on discord https://dvc.org/chat (I'm 'ruslan' there), so we could arrange something 🙂

Thanks,
Ruslan

dragonly · 2020-06-24T02:03:37Z

there's no docs about adding an oss remote, could you please add the corresponding instructions in the docs? thx

efiop · 2020-06-24T02:34:25Z

@dragonly There are: https://dvc.org/doc/command-reference/remote/add https://dvc.org/doc/command-reference/remote/modify

nanaya-tachibana mentioned this pull request May 2, 2019

add documentation for aliyun oss storage treeverse/dvc.org#285

Closed

efiop suggested changes May 2, 2019

View reviewed changes

nanaya-tachibana added a commit to nanaya-tachibana/dvc that referenced this pull request May 4, 2019

Not needed, since we don't support external dependencies/outputs on OSS.

f805712

See treeverse#1961.

nanaya-tachibana force-pushed the master branch 2 times, most recently from 098be92 to a8031ca Compare May 4, 2019 08:51

efiop suggested changes May 6, 2019

View reviewed changes

nanaya-tachibana force-pushed the master branch from 8190ccc to d030657 Compare May 8, 2019 01:33

nanaya-tachibana added 7 commits May 8, 2019 10:58

Not needed, since we don't support external dependencies/outputs on OSS.

341b0c6

See treeverse#1961.

Use default key id and key secret when they are not given,

740d059

which gives read access to public read bucket and public bucket.

test: add oss tests to appveyor.

a954a78

remove unneeded tests.

55efd59

nanaya-tachibana force-pushed the master branch from d030657 to 0c3a2ab Compare May 8, 2019 03:09

Ruslan Kuprieiev added 3 commits May 9, 2019 15:02

remote: fallback to [] if there are no cinfos

12bd313

Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai>

test: remote: add oss CLI test

ce74e2e

Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai>

travis: use iterative's fork of oss-emulator

47400bb

Just to keep things in-house. Signed-off-by: Ruslan Kuprieiev <ruslan@iterative.ai>

efiop merged commit 7ceaf88 into treeverse:master May 9, 2019

ghost reviewed May 10, 2019

View reviewed changes

dvc/remote/oss.py

class RemoteOSS(RemoteBase):

"""

oss2 document:

Copy link

ghost May 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😍

efiop mentioned this pull request May 13, 2019

remote: add support for aliyun oss #1954

Closed

skshetry mentioned this pull request Apr 27, 2022

fs.ossfs: remove unused var COPY_POLL_SECONDS #7647

Merged

remote: add support for aliyun oss #1961

remote: add support for aliyun oss #1961

Uh oh!

Conversation

nanaya-tachibana commented May 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

efiop commented May 2, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nanaya-tachibana commented May 4, 2019

Uh oh!

efiop left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

efiop commented May 7, 2019

Uh oh!

efiop commented May 7, 2019

Uh oh!

efiop commented May 9, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efiop commented May 10, 2019

Uh oh!

dragonly commented Jun 24, 2020

Uh oh!

efiop commented Jun 24, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nanaya-tachibana commented May 2, 2019 •

edited

Loading

efiop commented May 2, 2019 •

edited

Loading