how to download GRIT-20M dataset?

[stihuangyuan]

how to download GRIT-20M dataset?

[hhaAndroid]

In these few days, we will prepare the documents and thank you for your attention.

[lluo-Desktop]

In these few days, we will prepare the documents and thank you for your attention.

Hi @hhaAndroid
In GRIT-20M dataset download script, image size is 256(--image_size 256). This image size is samll to model input size. Do you have any suggests for download parameters?

img2dataset --url_list /path/to/GRIT_dataset/grit-20m --input_format "parquet"\
    --url_col "url" --caption_col "caption" --output_format webdataset \
    --output_folder /tmp/grit --processes_count 4 --thread_count 64 --image_size 256 \
    --resize_only_if_bigger=True --resize_mode="keep_ratio" --skip_reencode=True \
    --save_additional_columns '["id","noun_chunks","ref_exps","clip_similarity_vitb32","clip_similarity_vitl14"]' \
    --enable_wandb False