Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial #3745

TOsmanov · 2021-09-29T07:53:57Z

Motivation and context

Depended #3724

Issues:

To generate a manifest file, you need to download all the files to your pc that will be inconvenient for datasets of large volume. As well, adding data to the dataset will require updating the manifest file.
It may be worth adding support for S3 inventory in CVAT. In the S3 inventory contains paths to files with extension, and you can also add an additional field ETag which can be an MD5 digest of the object data.
After the file manifest is created, the name parameter has \\ Instead of /. For fix this, I used .replace in core.py file, perhaps there is a better solution.

Preview:
https://tosmanov.github.io/cvat/docs/manual/basics/attach-cloud-storage/

How has this been tested?

Checklist

I submit my changes into the develop branch
I have added description of my changes into CHANGELOG file
I have updated the documentation accordingly
I have added tests to cover my changes
I have linked related issues (read github docs)
I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below)

# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

Co-authored-by: Timur Osmanov <54434686+TOsmanov@users.noreply.github.com>

nmanovic · 2021-09-29T08:50:51Z

site/content/en/docs/manual/basics/attach-cloud-storage.md

+### Upload dataset
+
+Initially you need to create a manifest file for your image dataset. Information on how to do that is available
+on the [Simple command line to prepare dataset manifest file](/docs/manual/advanced/dataset_manifest) page.


@TOsmanov , let's repeat here basic steps to upload a simple dataset. Need exact instructions. Let's take a video file (take a link from serverless tutorial), prepare a manifest and upload them.

@nmanovic , as far as I understand cloud storage into UI does not support the use of video file, do you want me to describe the division of the video into frame sequence?

@Marishka17 , could you please comment on that?

@TOsmanov , OK. Let's use a dataset with images in the case.

Marishka17

Great work!

Marishka17 · 2021-10-05T09:04:49Z

site/content/en/docs/manual/basics/attach-cloud-storage.md

+[script for prepare manifest file](https://github.com/openvinotoolkit/cvat/tree/develop/utils/dataset_manifest).
+Perform the installation using the manual [aws-shell](https://github.com/awslabs/aws-shell),
+I used `aws-cli 1.20.49` `Python 3.7.9` `Windows 10`.
+You can configure credentials by running.


Suggested change

You can configure credentials by running.

You can configure credentials by running `aws configure`.

Marishka17 · 2021-10-05T09:04:55Z

site/content/en/docs/manual/basics/attach-cloud-storage.md

+![](/images/aws-s3_tutorial_7.jpg)
+
+### Prepare manifest file
+Now you need to prepare manifest file. I used [AWS cli](https://aws.amazon.com/cli/) and


Suggested change

Now you need to prepare manifest file. I used [AWS cli](https://aws.amazon.com/cli/) and

Now you need to prepare a manifest file. I used [AWS cli](https://aws.amazon.com/cli/) and

Marishka17 · 2021-10-05T09:05:02Z

site/content/en/docs/manual/basics/attach-cloud-storage.md

+### Create user and configure permissions
+
+To access bucket you will need to create a user, to do this, go [IAM](https://console.aws.amazon.com/iamv2/home#/users)
+and click `add users`. You need to choose AWS access type have access key ID and secret access key.


Suggested change

and click `add users`. You need to choose AWS access type have access key ID and secret access key.

and click `add users`. You need to choose AWS access type, have an access key ID and secret access key.

Marishka17 · 2021-10-05T09:05:06Z

site/content/en/docs/manual/basics/attach-cloud-storage.md

+You'll be taken to the bucket creation page. Here you have to specify the name of the bucket, region,
+optionally you can copy the settings of another bucket by clicking on the `choose bucket` button.
+Checkbox block all public access can be enabled as we will use `access key ID` and `secret access key` to gain access.
+In the following sections you can leave the default settings and click `create bucket`.


Suggested change

In the following sections you can leave the default settings and click `create bucket`.

In the following sections, you can leave the default settings and click `create bucket`.

Marishka17 · 2021-10-05T12:52:18Z

@TOsmanov , Could you please also prepare the same tutorial for Azure in another PR?

TOsmanov · 2021-10-05T13:16:31Z

@TOsmanov , Could you please also prepare the same tutorial for Azure in another PR?

@Marishka17 Okay, I'll add

nmanovic

Great job!

amotovx and others added 11 commits September 24, 2021 14:36

added info about cloud storage ui

d2d1806

fixed linter errors, improved list spacing

086dec0

fixed linter errors v2

0d60a1c

fixed mistakes & indentation

075a87e

Apply suggestions from code review

61dc224

Co-authored-by: Timur Osmanov <54434686+TOsmanov@users.noreply.github.com>

fixed errors & indentation issues

b15edf9

fixed errors & indentation issues v2

3e978d7

fix mistakes in attach-cloud-storage.md

413e042

apply suggestions v2

b34e1bd

Merge branch 'cloud-ui' of https://github.com/amotovx/cvat into cloud-ui

188bbc5

add aws-s3 tutorial

4af2ccd

TOsmanov requested review from azhavoro and nmanovic as code owners September 29, 2021 07:53

TOsmanov changed the title ~~Documentation update, added info about the cloud storage UI~~ Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial Sep 29, 2021

TOsmanov changed the title ~~Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial~~ [WIP]Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial Sep 29, 2021

TOsmanov added 3 commits September 29, 2021 11:03

fix linter errors

dbd631b

update attach-cloud-storage.md

8093114

fix mistakes

d99366c

nmanovic reviewed Sep 29, 2021

View reviewed changes

TOsmanov added 7 commits September 29, 2021 13:31

update attach-cloud-storage.md

e808e97

update image

ebdc35f

fix link

36df07e

update attach cloud storage section

72f2747

fix markup

33de118

fix mistake

ee7569b

fix mistake

ff8dce3

TOsmanov changed the title ~~[WIP]Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial~~ Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial Sep 30, 2021

TOsmanov added 2 commits October 4, 2021 13:52

Merge remote-tracking branch 'upstream/develop' into cloud-ui

d0f84c4

update the example manifest

9d7565f

nmanovic requested a review from Marishka17 October 5, 2021 07:31

Marishka17 reviewed Oct 5, 2021

View reviewed changes

apply suggestions

4b5edf7

Marishka17 approved these changes Oct 5, 2021

View reviewed changes

nmanovic approved these changes Oct 5, 2021

View reviewed changes

nmanovic merged commit ced1bc8 into cvat-ai:develop Oct 5, 2021

TOsmanov mentioned this pull request Oct 8, 2021

Add Azure Blob Container tutorial #3778

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial #3745

Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial #3745

TOsmanov commented Sep 29, 2021 •

edited

Loading

nmanovic Sep 29, 2021

TOsmanov Sep 29, 2021

nmanovic Sep 29, 2021

nmanovic Sep 30, 2021

Marishka17 left a comment

Marishka17 Oct 5, 2021

Marishka17 Oct 5, 2021

Marishka17 Oct 5, 2021

Marishka17 Oct 5, 2021

Marishka17 commented Oct 5, 2021

TOsmanov commented Oct 5, 2021

nmanovic left a comment

	You can configure credentials by running.
	You can configure credentials by running `aws configure`.

	Now you need to prepare manifest file. I used [AWS cli](https://aws.amazon.com/cli/) and
	Now you need to prepare a manifest file. I used [AWS cli](https://aws.amazon.com/cli/) and

	and click `add users`. You need to choose AWS access type have access key ID and secret access key.
	and click `add users`. You need to choose AWS access type, have an access key ID and secret access key.

	In the following sections you can leave the default settings and click `create bucket`.
	In the following sections, you can leave the default settings and click `create bucket`.

Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial #3745

Documentation update, added info about the cloud storage UI and add AWS-S3 tutorial #3745

Conversation

TOsmanov commented Sep 29, 2021 • edited Loading

Motivation and context

How has this been tested?

Checklist

License

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Marishka17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Marishka17 commented Oct 5, 2021

TOsmanov commented Oct 5, 2021

nmanovic left a comment

Choose a reason for hiding this comment

TOsmanov commented Sep 29, 2021 •

edited

Loading