Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mistakes in the GCC Dataset download commands #7

Open
PardoAlejo opened this issue Mar 30, 2022 · 1 comment
Open

Mistakes in the GCC Dataset download commands #7

PardoAlejo opened this issue Mar 30, 2022 · 1 comment

Comments

@PardoAlejo
Copy link

PardoAlejo commented Mar 30, 2022

Hi! I realized that the commands for downloading GCC 3M and 12M have a couple of typos. The corrected version for the 12M is below:

sed -i '1s/^/url\tcaption\n/' gcc12m.tsv
img2dataset --url_list gcc12m.tsv --input_format "tsv" \
            --url_col "url" --caption_col "caption" --output_format webdataset\
            --output_folder local_data/gcc12m_shards \
            --processes_count 16 --thread_count 64 \
            --image_size 512 --resize_mode keep_ratio --resize_only_if_bigger True \
            --enable_wandb True --save_metadata False --oom_shard_count 6

It would be nice if you can update the README.me with it.
Great work! Thank for sharing it.

@xvjiarui
Copy link
Contributor

Thx! Will do!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants