Downloading DataComp-1B

[linzhiqiu]

I am trying to download datacomp-1b, but it seems both my downloading speed and success rate are low, even after I set num_retries to 3:

worker - success: 0.887 - failed to download: 0.101 - failed to resize: 0.012 - images per sec: 8 - count: 10000
total - success: 0.883 - failed to download: 0.107 - failed to resize: 0.010 - images per sec: 103 - count: 247671
worker - success: 0.889 - failed to download: 0.101 - failed to resize: 0.010 - images per sec: 9 - count: 10000
total - success: 0.883 - failed to download: 0.107 - failed to resize: 0.010 - images per sec: 107 - count: 257671
worker - success: 0.894 - failed to download: 0.098 - failed to resize: 0.008 - images per sec: 8 - count: 10000
total - success: 0.883 - failed to download: 0.107 - failed to resize: 0.010 - images per sec: 111 - count: 267671
29it [40:28, 7.45s/it]worker - success: 0.890 - failed to download: 0.100 - failed to resize: 0.010 - images per sec: 8 - count: 10000
total - success: 0.884 - failed to download: 0.106 - failed to resize: 0.010 - images per sec: 115 - count: 277671
worker - success: 0.887 - failed to download: 0.103 - failed to resize: 0.010 - images per sec: 8 - count: 10000
total - success: 0.884 - failed to download: 0.106 - failed to resize: 0.010 - images per sec: 119 - count: 287671
worker - success: 0.891 - failed to download: 0.099 - failed to resize: 0.009 - images per sec: 8 - count: 10000
total - success: 0.884 - failed to download: 0.106 - failed to resize: 0.010 - images per sec: 123 - count: 297671
worker - success: 0.882 - failed to download: 0.108 - failed to resize: 0.010 - images per sec: 8 - count: 10000
total - success: 0.884 - failed to download: 0.106 - failed to resize: 0.010 - images per sec: 127 - count: 307671

[rom1504]

Hello, you need to setup a DNS resolver, see readme for details

…

On Sat, Jul 29, 2023, 07:35 linzhiqiu ***@***.***> wrote: I am trying to download datacomp-1b, but it seems both my downloading speed and success rate are low, even after I set num_retries to 3: 27it [40:23, 13.00s/it]worker - success: 0.887 - failed to download: 0.101 - failed to resize: 0.012 - images per sec: 8 - count: 10000 total - success: 0.883 - failed to download: 0.107 - failed to resize: 0.010 - images per sec: 103 - count: 247671 worker - success: 0.889 - failed to download: 0.101 - failed to resize: 0.010 - images per sec: 9 - count: 10000 total - success: 0.883 - failed to download: 0.107 - failed to resize: 0.010 - images per sec: 107 - count: 257671 worker - success: 0.894 - failed to download: 0.098 - failed to resize: 0.008 - images per sec: 8 - count: 10000 total - success: 0.883 - failed to download: 0.107 - failed to resize: 0.010 - images per sec: 111 - count: 267671 29it [40:28, 7.45s/it]worker - success: 0.890 - failed to download: 0.100 - failed to resize: 0.010 - images per sec: 8 - count: 10000 total - success: 0.884 - failed to download: 0.106 - failed to resize: 0.010 - images per sec: 115 - count: 277671 worker - success: 0.887 - failed to download: 0.103 - failed to resize: 0.010 - images per sec: 8 - count: 10000 total - success: 0.884 - failed to download: 0.106 - failed to resize: 0.010 - images per sec: 119 - count: 287671 worker - success: 0.891 - failed to download: 0.099 - failed to resize: 0.009 - images per sec: 8 - count: 10000 total - success: 0.884 - failed to download: 0.106 - failed to resize: 0.010 - images per sec: 123 - count: 297671 worker - success: 0.882 - failed to download: 0.108 - failed to resize: 0.010 - images per sec: 8 - count: 10000 total - success: 0.884 - failed to download: 0.106 - failed to resize: 0.010 - images per sec: 127 - count: 307671 — Reply to this email directly, view it on GitHub <#41>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAR437VU6VKFPA6UAVLRGPDXSSOL3ANCNFSM6AAAAAA24I7SCE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>