recipe for using the Lightly Docker as API worker

[MalteEbner]

Description

• links to the docs of setting up the S3 bucket

• links to the docs of installing the docker

• links to the docs of the first steps with the docker

• changes the tutorial of setting up the S3 bucket to include distinguishing between Images/Videos

Generated report

Using the docker with an S3 bucket as remote datasource. — lightly 1.2.3 documentation.pdf

[codecov]

Codecov Report

Merging #630 (9a51601) into master (deef6be) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #630   +/-   ##
=======================================
  Coverage   87.70%   87.70%           
=======================================
  Files          89       89           
  Lines        3433     3433           
=======================================
  Hits         3011     3011           
  Misses        422      422           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update deef6be...9a51601. Read the comment docs.

[japrescott]

some comments. I am not sure if the title of the tutorial is correct as this tutorial should be more about directly using a datasource instead of needing the data locally

[philippmwirth]

Just had a quick look and the "setting up" is too minimal and is missing a few points:

• there should be a case distinction between video datasets and image datasets
  -> if you want to process images, choose Image when creating the dataset
  -> if you want to process videos, choose Video when creating the dataset

[MalteEbner]

Just had a quick look and the "setting up" is too minimal and is missing a few points:

• there should be a case distinction between video datasets and image datasets
  -> if you want to process images, choose Image when creating the dataset
  -> if you want to process videos, choose Video when creating the dataset

The setting up of the S3 datasource is covered in the dataset creation with AWS recipe: https://docs.lightly.ai/getting_started/dataset_creation/dataset_creation_aws_bucket.html
It surely belongs there and not into the docker tutorial.

[philippmwirth]

Just had a quick look and the "setting up" is too minimal and is missing a few points:

• there should be a case distinction between video datasets and image datasets
  -> if you want to process images, choose Image when creating the dataset
  -> if you want to process videos, choose Video when creating the dataset

The setting up of the S3 datasource is covered in the dataset creation with AWS recipe: https://docs.lightly.ai/getting_started/dataset_creation/dataset_creation_aws_bucket.html It surely belongs there and not into the docker tutorial.

All I'm saying is that if I follow the tutorial now it will not work as expected.

[MalteEbner]

All I'm saying is that if I follow the tutorial now it will not work as expected.

I adapted the tutorial of setting up the S3 bucket accordingly.

[IgorSusmelj]

I left a few comments. It worked well and is CRAZY SUPER COOL!!! :D

No, it's really awesome to see that this is working. Sooooo convenient.

Could you also add a line here on top? https://docs.lightly.ai/docker/overview.html
Maybe mark the active learning part not as new anymore.

[IgorSusmelj]

I would assume at this part that there is an S3 bucket. The user should go to the other link to setup s3 properly or have a look if there are any questions.

However, for the sake of this tutorial, I would suggest adding all the elements starting from the dataset creation in the UI here as well. So we guide the user from creating a dataset in Lightly and connecting it with the S3 bucket to running the docker.

[IgorSusmelj]

Please also tell the user here that he/she needs to pick images or videos depending on the dataset type he/she wants to use

[MalteEbner]

I would not add all the dataset creation elements here for two reasons:

• It would mean duplicating lots of stuff.
• It does not work well when we also have support for GC storage and azure storage.

[IgorSusmelj]

Would add a screenshot here of the UI home screen for the dataset (e.g. how should it look like)

[IgorSusmelj]

This is sort of the killer feature that appears in the end and it goes under because it's just lots of text. I would try to use bullet points or images/ screenshots to help. E.g.

Running the docker for the first time:

Running the docker again with a new video (we see a new cluster because the new video is very different):

Only visualizing the newly added data:

[MalteEbner]

I added much more content and screenshots.

[IgorSusmelj]

Ah, and one other thing:

• we should mention that this only works with S3 and when to expect other sources (azure, gcp) to work as well

[MalteEbner]

we should mention that this only works with S3 and when to expect other sources (azure, gcp) to work as well

I added which features don't work yet but will be implemented soon.

[IgorSusmelj]

I would try to give a more complete docker run command. Maybe add the stopping condition?

[IgorSusmelj]

Thanks for the changes. It already looks much better. I would still add a more detailed instruction on the s3 setup. Just mention the steps:

1. create dataset (use videos or images depending on the data)
2. edit dataset and click on s3
3. fill out (here I would add a screenshot, we can even recycle the one we already have :))

The reason why I would add this is to give the user a good end-to-end workflow. Otherwise, he/she has to jump around in our docs and search the various parts together.

[MalteEbner]

Thanks for the changes. It already looks much better. I would still add a more detailed instruction on the s3 setup. Just mention the steps:

1. create dataset (use videos or images depending on the data)
2. edit dataset and click on s3
3. fill out (here I would add a screenshot, we can even recycle the one we already have :))

The reason why I would add this is to give the user a good end-to-end workflow. Otherwise, he/she has to jump around in our docs and search the various parts together.

I have some questions regarding your proposal:

• The 3 steps you proposed are basically the 6 steps in the section "Create and configure a dataset" from the docs: https://docs.lightly.ai/getting_started/dataset_creation/dataset_creation_aws_bucket.html#uploading-your-data
  Do I get that correctly that you would leave away the complete first section of Setting up Amazon S3? Would you link it? Or how would you tell the user what the access key and secret access key are?
• How would you extend the documentation when we add GC bucket and azure storage? Create new tutorials for them? Or extend this tutorial?
• Would you also try to get rid of the other references to the docs? (How to download the docker, the first steps with the docker)

However, I get that the S3 tutorial is not ideal for that, thus I created a new issue: https://linear.app/lightly/issue/LIG-548/datasource-setup-recipes-use-order-fitting-milestone-0