ValueError: too many file descriptors in select()

[friggog]

Windows with at least DVC>=2.6.0 (perhaps earlier versions) and azure remote
I have large number of directories (say 200) each with say 10 files in, the directories are tracked directly with DVC.
I'm trying to run dvc pull to download these and I get ValueError: too many file descriptors in select() and CMD hangs forever.

Another related but non-critical issue is that DVC seems to query the remote separately for every *.dir file it needs to download and download them individually which is really slow - can't the remote be queried once and all ~200 dir files be downloaded at once?

>dvc pull -v
2021-09-15 10:54:54,190 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:54:54,192 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:54:54,193 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:54:54,197 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:54:54,198 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:54:54,297 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:55:36,775 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:36,863 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:37,225 DEBUG: Downloading 'azure://my-remote/db/b51231e80a0dbd0f1fe5786242c766.dir' to '.dvc\cache\db\b51231e80a0dbd0f1fe5786242c766.dir'
2021-09-15 10:55:37,376 DEBUG: state save (2912436261517855907, 1631699737330944256, 603) dbb51231e80a0dbd0f1fe5786242c766.dir
2021-09-15 10:55:37,454 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:55:37,457 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:55:37,460 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:55:37,465 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:55:37,466 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:55:37,570 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:55:39,880 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:39,944 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:40,349 DEBUG: Downloading 'azure://my-remote/39/1169727bee5ec91ef388cd0004917c.dir' to '.dvc\cache\39\1169727bee5ec91ef388cd0004917c.dir'
2021-09-15 10:55:40,484 DEBUG: state save (1881485164717332997, 1631699740477027584, 603) 391169727bee5ec91ef388cd0004917c.dir
2021-09-15 10:55:40,578 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:55:40,580 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:55:40,582 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:55:40,588 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:55:40,589 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:55:40,717 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:55:42,733 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:42,804 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:43,293 DEBUG: Downloading 'azure://my-remote/85/1c6738d144c116f8bc9dc8595a3a5a.dir' to '.dvc\cache\85\1c6738d144c116f8bc9dc8595a3a5a.dir'
2021-09-15 10:55:43,414 DEBUG: state save (2664473120733366070, 1631699743407263488, 603) 851c6738d144c116f8bc9dc8595a3a5a.dir
2021-09-15 10:55:43,522 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:55:43,525 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:55:43,527 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:55:43,535 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:55:43,537 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:55:43,643 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:55:47,130 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:47,194 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:47,722 DEBUG: Downloading 'azure://my-remote/d9/1dc90286ad34727eda0faffb9c0529.dir' to '.dvc\cache\d9\1dc90286ad34727eda0faffb9c0529.dir'
2021-09-15 10:55:47,827 DEBUG: state save (2664473120739191315, 1631699747821704704, 603) d91dc90286ad34727eda0faffb9c0529.dir
2021-09-15 10:55:47,894 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:55:47,896 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:55:47,898 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:55:47,902 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:55:47,903 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:55:47,998 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:55:51,779 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:51,853 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:52,267 DEBUG: Downloading 'azure://my-remote/f1/32b5218127af55aec353761c252dee.dir' to '.dvc\cache\f1\32b5218127af55aec353761c252dee.dir'
2021-09-15 10:55:52,376 DEBUG: state save (7103619552507012856, 1631699752370092288, 603) f132b5218127af55aec353761c252dee.dir
2021-09-15 10:55:52,467 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:55:52,469 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:55:52,471 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:55:52,476 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:55:52,477 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:55:52,595 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:55:54,545 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:54,624 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:55,109 DEBUG: Downloading 'azure://my-remote/05/6d7e6c19902301fd3891c864b94ee0.dir' to '.dvc\cache\05\6d7e6c19902301fd3891c864b94ee0.dir'
2021-09-15 10:55:55,229 DEBUG: state save (3904281985119155855, 1631699755222837248, 603) 056d7e6c19902301fd3891c864b94ee0.dir
2021-09-15 10:55:55,301 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:55:55,303 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:55:55,305 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:55:55,310 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:55:55,311 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:55:55,415 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:55:57,894 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:57,969 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:55:58,476 DEBUG: Downloading 'azure://my-remote/99/461c4482257e0a2da7420844a28f92.dir' to '.dvc\cache\99\461c4482257e0a2da7420844a28f92.dir'
2021-09-15 10:55:58,594 DEBUG: state save (3121291592579061681, 1631699758587884544, 603) 99461c4482257e0a2da7420844a28f92.dir
2021-09-15 10:55:58,693 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:55:58,695 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:55:58,698 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:55:58,702 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:55:58,704 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:55:58,825 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:56:00,662 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:00,731 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:01,263 DEBUG: Downloading 'azure://my-remote/a7/051ea171af7508476638e2f0e4f588.dir' to '.dvc\cache\a7\051ea171af7508476638e2f0e4f588.dir'
2021-09-15 10:56:01,395 DEBUG: state save (6053281630900451211, 1631699761387491584, 603) a7051ea171af7508476638e2f0e4f588.dir
2021-09-15 10:56:01,495 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:56:01,497 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:56:01,500 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:56:01,506 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:56:01,507 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:56:01,639 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:56:04,192 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:04,268 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:04,829 DEBUG: Downloading 'azure://my-remote/11/5a245027722b301a2056ecc9849b4b.dir' to '.dvc\cache\11\5a245027722b301a2056ecc9849b4b.dir'
2021-09-15 10:56:05,001 DEBUG: state save (2333858505471818690, 1631699764994374400, 603) 115a245027722b301a2056ecc9849b4b.dir
2021-09-15 10:56:05,107 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:56:05,110 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:56:05,112 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:56:05,118 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:56:05,120 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:56:05,231 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:56:07,172 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:07,250 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:07,759 DEBUG: Downloading 'azure://my-remote/78/eaf8fff631709ae58eb11aeacf0fe1.dir' to '.dvc\cache\78\eaf8fff631709ae58eb11aeacf0fe1.dir'
2021-09-15 10:56:07,876 DEBUG: state save (2333858505498614817, 1631699767869417216, 603) 78eaf8fff631709ae58eb11aeacf0fe1.dir
2021-09-15 10:56:07,985 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:56:07,986 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:56:07,989 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:56:07,995 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:56:07,996 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:56:08,098 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:56:10,331 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:10,396 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:10,824 DEBUG: Downloading 'azure://my-remote/a8/85916d3638d54407b8e61b2267daff.dir' to '.dvc\cache\a8\85916d3638d54407b8e61b2267daff.dir'
2021-09-15 10:56:10,963 DEBUG: state save (1672630496652891751, 1631699770957171456, 603) a885916d3638d54407b8e61b2267daff.dir
2021-09-15 10:56:11,020 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:56:11,022 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:56:11,023 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:56:11,027 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:56:11,028 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:56:11,117 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:56:13,739 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:13,814 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:14,280 DEBUG: Downloading 'azure://my-remote/4f/d0d2a6ede96cf455b261ac5fb998e2.dir' to '.dvc\cache\4f\d0d2a6ede96cf455b261ac5fb998e2.dir'
2021-09-15 10:56:14,402 DEBUG: state save (2581820230443326816, 1631699774395614208, 603) 4fd0d2a6ede96cf455b261ac5fb998e2.dir
2021-09-15 10:56:14,494 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:56:14,496 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:56:14,498 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:56:14,503 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:56:14,505 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:56:14,620 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:56:18,920 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:18,995 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:19,469 DEBUG: Downloading 'azure://my-remote/d7/309d4d492f82ae93cb89f5c6f81990.dir' to '.dvc\cache\d7\309d4d492f82ae93cb89f5c6f81990.dir'
2021-09-15 10:56:19,597 DEBUG: state save (2333858505485799278, 1631699779590052608, 603) d7309d4d492f82ae93cb89f5c6f81990.dir
2021-09-15 10:56:19,703 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:56:19,705 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:56:19,707 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:56:19,714 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:56:19,715 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:56:19,818 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:56:22,260 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:22,332 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:22,808 DEBUG: Downloading 'azure://my-remote/03/f6ceb80a8279c2cbb8c15de4ac113d.dir' to '.dvc\cache\03\f6ceb80a8279c2cbb8c15de4ac113d.dir'
2021-09-15 10:56:22,917 DEBUG: state save (1755284303052416028, 1631699782911220992, 603) 03f6ceb80a8279c2cbb8c15de4ac113d.dir
2021-09-15 10:56:23,024 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:56:23,027 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:56:23,030 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:56:23,036 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:56:23,037 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:56:23,141 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:56:25,200 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:25,273 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:25,739 DEBUG: Downloading 'azure://my-remote/54/e4604be059efcb5efddaaec038c1d3.dir' to '.dvc\cache\54\e4604be059efcb5efddaaec038c1d3.dir'
2021-09-15 10:56:25,854 DEBUG: state save (846091016842337172, 1631699785848288256, 603) 54e4604be059efcb5efddaaec038c1d3.dir
2021-09-15 10:56:25,958 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:56:25,961 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:56:25,963 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:56:25,969 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:56:25,970 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:56:26,096 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:56:28,178 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:28,255 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:28,721 DEBUG: Downloading 'azure://my-remote/d3/6c623c5a9636efdfd0bf9ed55223a9.dir' to '.dvc\cache\d3\6c623c5a9636efdfd0bf9ed55223a9.dir'
2021-09-15 10:56:28,841 DEBUG: state save (2581820230444491865, 1631699788833970688, 603) d36c623c5a9636efdfd0bf9ed55223a9.dir
2021-09-15 10:56:28,943 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:56:28,945 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:56:28,947 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:56:28,954 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:56:28,955 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:56:29,059 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:56:30,883 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:30,961 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:56:31,476 DEBUG: Downloading 'azure://my-remote/33/7f31e963911b2e28b50fa814510228.dir' to '.dvc\cache\33\7f31e963911b2e28b50fa814510228.dir'
2021-09-15 10:56:31,603 DEBUG: state save (2581820230325656867, 1631699791597101568, 603) 337f31e963911b2e28b50fa814510228.dir
2021-09-15 10:56:31,703 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:56:31,706 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:56:31,708 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:56:31,715 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:56:31,716 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:56:31,824 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:57:13,260 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:57:13,338 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:57:13,700 DEBUG: Downloading 'azure://my-remote/ca/f884d145de4c7fb865b2a1da5a6d4c.dir' to '.dvc\cache\ca\f884d145de4c7fb865b2a1da5a6d4c.dir'
2021-09-15 10:57:13,815 DEBUG: state save (2475006299367402635, 1631699833808065536, 603) caf884d145de4c7fb865b2a1da5a6d4c.dir
2021-09-15 10:57:13,917 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:57:13,919 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:57:13,920 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:57:13,925 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:57:13,928 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:57:14,012 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:58:35,016 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:58:35,100 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:58:35,451 DEBUG: Downloading 'azure://my-remote/60/473b8c82a39d9d54639b35b812312c.dir' to '.dvc\cache\60\473b8c82a39d9d54639b35b812312c.dir'
2021-09-15 10:58:35,570 DEBUG: state save (3986935207190684280, 1631699915563839488, 603) 60473b8c82a39d9d54639b35b812312c.dir
2021-09-15 10:58:35,659 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:58:35,661 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:58:35,665 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:58:35,670 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:58:35,671 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:58:35,757 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:16,493 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:16,568 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:16,995 DEBUG: Downloading 'azure://my-remote/ef/09bee90c2730c82ac9a8befe0ae8d9.dir' to '.dvc\cache\ef\09bee90c2730c82ac9a8befe0ae8d9.dir'
2021-09-15 10:59:17,117 DEBUG: state save (3077744340208349071, 1631699957110445056, 603) ef09bee90c2730c82ac9a8befe0ae8d9.dir
2021-09-15 10:59:17,224 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:17,226 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:17,228 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:17,234 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:17,235 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:17,324 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:19,167 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:19,239 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:19,665 DEBUG: Downloading 'azure://my-remote/6d/8237994f47b0484e6127651c1515d0.dir' to '.dvc\cache\6d\8237994f47b0484e6127651c1515d0.dir'
2021-09-15 10:59:19,787 DEBUG: state save (2995088957491702549, 1631699959780902144, 603) 6d8237994f47b0484e6127651c1515d0.dir
2021-09-15 10:59:19,891 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:19,893 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:19,896 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:19,901 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:19,902 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:19,988 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:22,084 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:22,156 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:22,701 DEBUG: Downloading 'azure://my-remote/64/4e2ec0cf7d1e65f9220b6a196d2fb8.dir' to '.dvc\cache\64\4e2ec0cf7d1e65f9220b6a196d2fb8.dir'
2021-09-15 10:59:22,815 DEBUG: state save (4813473565748760867, 1631699962806881280, 603) 644e2ec0cf7d1e65f9220b6a196d2fb8.dir
2021-09-15 10:59:22,899 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:22,900 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:22,902 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:22,905 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:22,907 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:23,025 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:25,958 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:26,016 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:26,472 DEBUG: Downloading 'azure://my-remote/c5/9f75077d7acf285d29aed1b0f9334d.dir' to '.dvc\cache\c5\9f75077d7acf285d29aed1b0f9334d.dir'
2021-09-15 10:59:26,593 DEBUG: state save (1755284303036105342, 1631699966583963648, 603) c59f75077d7acf285d29aed1b0f9334d.dir
2021-09-15 10:59:26,683 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:26,685 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:26,687 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:26,693 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:26,694 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:26,802 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:29,461 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:29,534 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:30,068 DEBUG: Downloading 'azure://my-remote/65/bc354e70d30f4866bfc8a0d6750dab.dir' to '.dvc\cache\65\bc354e70d30f4866bfc8a0d6750dab.dir'
2021-09-15 10:59:30,187 DEBUG: state save (2581820230344297651, 1631699970180367616, 603) 65bc354e70d30f4866bfc8a0d6750dab.dir
2021-09-15 10:59:30,290 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:30,293 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:30,296 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:30,305 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:30,306 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:30,403 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:33,466 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:33,537 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:34,030 DEBUG: Downloading 'azure://my-remote/d2/7738c6447163c6bd0a13ed81b0e386.dir' to '.dvc\cache\d2\7738c6447163c6bd0a13ed81b0e386.dir'
2021-09-15 10:59:34,515 DEBUG: state save (2581820230340802504, 1631699974506218240, 603) d27738c6447163c6bd0a13ed81b0e386.dir
2021-09-15 10:59:34,611 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:34,613 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:34,616 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:34,624 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:34,626 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:34,755 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:37,321 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:37,393 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:37,827 DEBUG: Downloading 'azure://my-remote/b1/213bc6b64940a26a96e8d5df9dbf18.dir' to '.dvc\cache\b1\213bc6b64940a26a96e8d5df9dbf18.dir'
2021-09-15 10:59:37,947 DEBUG: state save (5367889193221302641, 1631699977939902976, 603) b1213bc6b64940a26a96e8d5df9dbf18.dir
2021-09-15 10:59:38,052 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:38,055 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:38,057 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:38,063 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:38,064 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:38,166 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:40,251 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:40,325 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:40,707 DEBUG: Downloading 'azure://my-remote/f5/9ee58b42029535211ca7758fc02681.dir' to '.dvc\cache\f5\9ee58b42029535211ca7758fc02681.dir'
2021-09-15 10:59:40,826 DEBUG: state save (2747126927141045690, 1631699980819972096, 603) f59ee58b42029535211ca7758fc02681.dir
2021-09-15 10:59:40,920 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:40,923 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:40,925 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:40,931 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:40,933 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:41,026 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:43,252 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:43,322 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:43,865 DEBUG: Downloading 'azure://my-remote/43/92b922ae627318c210d47af8492dd7.dir' to '.dvc\cache\43\92b922ae627318c210d47af8492dd7.dir'
2021-09-15 10:59:43,985 DEBUG: state save (3243050682512142733, 1631699983978857472, 603) 4392b922ae627318c210d47af8492dd7.dir
2021-09-15 10:59:44,083 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:44,085 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:44,088 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:44,094 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:44,098 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:44,182 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:46,380 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:46,457 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:46,943 DEBUG: Downloading 'azure://my-remote/b4/a498637ff04e3a9c60f9cc416b014f.dir' to '.dvc\cache\b4\a498637ff04e3a9c60f9cc416b014f.dir'
2021-09-15 10:59:47,059 DEBUG: state save (2829781039059524580, 1631699987053569792, 603) b4a498637ff04e3a9c60f9cc416b014f.dir
2021-09-15 10:59:47,142 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:47,144 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:47,145 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:47,150 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:47,151 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:47,241 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:49,403 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:49,478 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:49,949 DEBUG: Downloading 'azure://my-remote/40/5c9cd1e6983b64fbd9092fefde873b.dir' to '.dvc\cache\40\5c9cd1e6983b64fbd9092fefde873b.dir'
2021-09-15 10:59:50,073 DEBUG: state save (2747126927144540837, 1631699990065082368, 603) 405c9cd1e6983b64fbd9092fefde873b.dir
2021-09-15 10:59:50,179 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:50,181 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:50,183 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:50,190 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:50,191 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:50,278 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:52,754 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:52,834 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:53,245 DEBUG: Downloading 'azure://my-remote/fe/7576f34edf661a64fdec39859d73e1.dir' to '.dvc\cache\fe\7576f34edf661a64fdec39859d73e1.dir'
2021-09-15 10:59:53,362 DEBUG: state save (2829781039057194482, 1631699993355135232, 603) fe7576f34edf661a64fdec39859d73e1.dir
2021-09-15 10:59:53,457 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:53,460 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:53,462 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:53,467 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:53,469 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:53,553 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:55,661 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:55,733 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:56,369 DEBUG: Downloading 'azure://my-remote/9a/5f58a954a2f4f3c6c6a92e836fcc11.dir' to '.dvc\cache\9a\5f58a954a2f4f3c6c6a92e836fcc11.dir'
2021-09-15 10:59:56,485 DEBUG: state save (2581820230334977259, 1631699996478923520, 603) 9a5f58a954a2f4f3c6c6a92e836fcc11.dir
2021-09-15 10:59:56,595 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:56,597 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:56,600 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:56,606 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:56,607 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:56,691 DEBUG: Querying 230 hashes via object_exists
2021-09-15 10:59:58,854 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:58,916 DEBUG: Querying 1 hashes via object_exists
2021-09-15 10:59:59,483 DEBUG: Downloading 'azure://my-remote/ae/3f4fa3bced5b5e1c546c53400d6ef2.dir' to '.dvc\cache\ae\3f4fa3bced5b5e1c546c53400d6ef2.dir'
2021-09-15 10:59:59,605 DEBUG: state save (2829781039060689629, 1631699999597140480, 603) ae3f4fa3bced5b5e1c546c53400d6ef2.dir
2021-09-15 10:59:59,688 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 10:59:59,690 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 10:59:59,693 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 10:59:59,698 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 10:59:59,700 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 10:59:59,786 DEBUG: Querying 230 hashes via object_exists
2021-09-15 11:00:02,419 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:02,496 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:02,984 DEBUG: Downloading 'azure://my-remote/6d/ca5a39e1b5a14e60474a0e5bdf72ca.dir' to '.dvc\cache\6d\ca5a39e1b5a14e60474a0e5bdf72ca.dir'
2021-09-15 11:00:03,101 DEBUG: state save (1755284303054746126, 1631700003096579840, 603) 6dca5a39e1b5a14e60474a0e5bdf72ca.dir
2021-09-15 11:00:03,186 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 11:00:03,188 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 11:00:03,190 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 11:00:03,195 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 11:00:03,198 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 11:00:03,284 DEBUG: Querying 230 hashes via object_exists
2021-09-15 11:00:05,364 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:05,431 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:05,903 DEBUG: Downloading 'azure://my-remote/cf/8b5dfe8dd0a460e6aa3d8701441620.dir' to '.dvc\cache\cf\8b5dfe8dd0a460e6aa3d8701441620.dir'
2021-09-15 11:00:06,031 DEBUG: state save (2664473120613366023, 1631700006024068352, 603) cf8b5dfe8dd0a460e6aa3d8701441620.dir
2021-09-15 11:00:06,131 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 11:00:06,134 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 11:00:06,136 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 11:00:06,142 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 11:00:06,143 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 11:00:06,236 DEBUG: Querying 230 hashes via object_exists
2021-09-15 11:00:08,548 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:08,620 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:09,131 DEBUG: Downloading 'azure://my-remote/1f/3380373b1942c8af2839219180a0fd.dir' to '.dvc\cache\1f\3380373b1942c8af2839219180a0fd.dir'
2021-09-15 11:00:09,252 DEBUG: state save (2581820230364103484, 1631700009246375680, 603) 1f3380373b1942c8af2839219180a0fd.dir
2021-09-15 11:00:09,345 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 11:00:09,347 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 11:00:09,350 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 11:00:09,355 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 11:00:09,356 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 11:00:09,452 DEBUG: Querying 230 hashes via object_exists
2021-09-15 11:00:12,460 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:12,555 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:13,201 DEBUG: Downloading 'azure://my-remote/22/c0f4bb8bee735ceecc32177c45344a.dir' to '.dvc\cache\22\c0f4bb8bee735ceecc32177c45344a.dir'
2021-09-15 11:00:13,327 DEBUG: state save (2747126927137550543, 1631700013320934144, 603) 22c0f4bb8bee735ceecc32177c45344a.dir
2021-09-15 11:00:13,411 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 11:00:13,413 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 11:00:13,415 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 11:00:13,419 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 11:00:13,421 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 11:00:13,516 DEBUG: Querying 230 hashes via object_exists
2021-09-15 11:00:19,441 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:19,512 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:19,970 DEBUG: Downloading 'azure://my-remote/2a/06bac5b388a8014131037d220f64df.dir' to '.dvc\cache\2a\06bac5b388a8014131037d220f64df.dir'
2021-09-15 11:00:20,103 DEBUG: state save (2581820230361773386, 1631700020096268544, 603) 2a06bac5b388a8014131037d220f64df.dir
2021-09-15 11:00:20,202 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 11:00:20,205 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 11:00:20,209 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 11:00:20,215 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 11:00:20,216 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 11:00:20,327 DEBUG: Querying 230 hashes via object_exists
2021-09-15 11:00:23,389 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:23,460 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:23,890 DEBUG: Downloading 'azure://my-remote/d5/45853d3814c7a8d8664d711114ac92.dir' to '.dvc\cache\d5\45853d3814c7a8d8664d711114ac92.dir'
2021-09-15 11:00:24,011 DEBUG: state save (2581820230362938435, 1631700024004060672, 603) d545853d3814c7a8d8664d711114ac92.dir
2021-09-15 11:00:24,088 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 11:00:24,090 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 11:00:24,092 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 11:00:24,096 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 11:00:24,097 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 11:00:24,211 DEBUG: Querying 230 hashes via object_exists
2021-09-15 11:00:28,645 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:28,723 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:29,248 DEBUG: Downloading 'azure://my-remote/c2/85769d49c3d064c25a5f9cb125e744.dir' to '.dvc\cache\c2\85769d49c3d064c25a5f9cb125e744.dir'
2021-09-15 11:00:29,362 DEBUG: state save (2829781039088650805, 1631700029355005952, 603) c285769d49c3d064c25a5f9cb125e744.dir
2021-09-15 11:00:29,454 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 11:00:29,457 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 11:00:29,460 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 11:00:29,464 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 11:00:29,465 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 11:00:29,595 DEBUG: Querying 230 hashes via object_exists
2021-09-15 11:00:32,232 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:32,306 DEBUG: Querying 1 hashes via object_exists
2021-09-15 11:00:32,830 DEBUG: Downloading 'azure://my-remote/eb/fc4729c7e7e13a2decb64103967e56.dir' to '.dvc\cache\eb\fc4729c7e7e13a2decb64103967e56.dir'
2021-09-15 11:00:32,971 DEBUG: state save (2581820230360608337, 1631700032963369472, 603) ebfc4729c7e7e13a2decb64103967e56.dir
2021-09-15 11:00:33,055 DEBUG: Preparing to transfer data from 'azure://my-remote' to '.dvc\cache'
2021-09-15 11:00:33,057 DEBUG: Preparing to collect status from '.dvc\cache'
2021-09-15 11:00:33,060 DEBUG: Collecting status from '.dvc\cache'
2021-09-15 11:00:33,065 DEBUG: Preparing to collect status from 'azure://my-remote'
2021-09-15 11:00:33,066 DEBUG: Collecting status from 'azure://my-remote'
2021-09-15 11:00:33,183 DEBUG: Querying 230 hashes via object_exists
  0% Querying remote cache|                                                                    |0/230 [00:00<?,     ?file/s]Exception in thread fsspecIO:
Traceback (most recent call last):
  File "...\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "...\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "...\lib\asyncio\base_events.py", line 541, in run_forever
    self._run_once()
  File "...\lib\asyncio\base_events.py", line 1750, in _run_once
    event_list = self._selector.select(timeout)
  File "...\lib\selectors.py", line 323, in select
    r, w, _ = self._select(self._readers, self._writers, [], timeout)
  File "...\lib\selectors.py", line 314, in _select
    r, w, x = select.select(r, w, w, timeout)
ValueError: too many file descriptors in select()

[friggog]

The exception is resolved in (or before) DVC 2.8.1, the performance comment still stands though

[efiop]

@friggog THanks for the feedback! The fetching will be addressed in the future, we are working on pre-requisites right now.

[friggog]

I'm still hitting this exception in latest release btw - seems like it was not resolved previously

[efiop]

@friggog Please post verbose log with the newest dvc version.

[friggog]

([conda_env]) ...\dvc_dir>dvc pull -v
2022-07-18 13:56:35,134 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:56:35,137 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:56:35,139 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:56:35,146 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:56:35,147 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:56:35,329 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:56:38,839 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:38,898 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:39,231 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:56:39,233 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:56:39,235 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:56:39,239 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:56:39,240 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:56:39,396 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:56:40,975 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:41,038 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:41,397 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:56:41,399 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:56:41,401 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:56:41,405 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:56:41,406 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:56:41,560 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:56:42,764 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:42,835 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:43,214 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:56:43,216 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:56:43,217 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:56:43,221 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:56:43,222 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:56:43,384 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:56:44,563 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:44,635 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:44,940 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:56:44,942 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:56:44,943 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:56:44,948 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:56:44,949 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:56:45,106 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:56:46,331 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:46,379 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:46,714 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:56:46,716 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:56:46,717 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:56:46,722 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:56:46,723 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:56:46,887 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:56:48,081 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:48,267 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:48,620 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:56:48,622 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:56:48,624 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:56:48,628 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:56:48,630 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:56:48,791 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:56:50,035 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:50,093 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:50,490 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:56:50,492 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:56:50,494 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:56:50,498 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:56:50,499 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:56:50,670 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:56:51,877 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:51,944 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:52,291 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:56:52,294 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:56:52,295 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:56:52,299 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:56:52,301 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:56:52,492 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:56:53,616 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:53,685 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:54,033 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:56:54,035 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:56:54,036 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:56:54,040 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:56:54,041 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:56:54,198 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:56:55,471 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:55,540 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:55,870 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:56:55,872 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:56:55,874 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:56:55,878 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:56:55,880 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:56:56,035 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:56:57,297 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:57,370 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:57,683 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:56:57,685 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:56:57,687 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:56:57,691 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:56:57,692 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:56:57,872 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:56:59,051 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:59,104 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:56:59,471 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:56:59,473 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:56:59,474 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:56:59,478 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:56:59,480 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:56:59,644 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:57:00,770 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:00,829 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:01,185 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:57:01,187 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:57:01,188 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:57:01,193 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:57:01,194 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:57:01,368 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:57:02,591 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:02,645 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:03,001 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:57:03,003 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:57:03,005 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:57:03,008 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:57:03,009 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:57:03,155 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:57:04,425 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:04,489 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:04,834 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:57:04,835 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:57:04,836 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:57:04,839 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:57:04,840 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:57:05,008 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:57:06,361 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:06,438 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:06,791 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:57:06,793 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:57:06,794 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:57:06,798 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:57:06,800 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:57:06,989 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:57:08,206 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:08,260 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:08,733 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:57:08,734 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:57:08,736 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:57:08,740 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:57:08,741 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:57:08,901 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:57:10,021 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:10,077 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:10,502 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:57:10,504 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:57:10,505 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:57:10,509 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:57:10,511 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:57:10,661 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:57:11,926 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:11,975 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:12,400 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:57:12,401 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:57:12,403 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:57:12,407 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:57:12,408 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:57:12,594 DEBUG: Querying 254 oids via object_exists
2022-07-18 13:57:13,866 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:13,924 DEBUG: Querying 1 oids via object_exists
2022-07-18 13:57:14,295 DEBUG: Preparing to transfer data from '[container_name]' to '[local_cache]'
2022-07-18 13:57:14,297 DEBUG: Preparing to collect status from '[local_cache]'
2022-07-18 13:57:14,298 DEBUG: Collecting status from '[local_cache]'
2022-07-18 13:57:14,302 DEBUG: Preparing to collect status from '[container_name]'
2022-07-18 13:57:14,304 DEBUG: Collecting status from '[container_name]'
2022-07-18 13:57:14,477 DEBUG: Querying 254 oids via object_exists
  0% Querying remote cache|                                                                                                                                   |0/254 [00:00<?,    ?files/sE 
xception in thread fsspecIO:
Traceback (most recent call last):
  File "...\Anaconda3\envs\[conda_env]\lib\threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "...\Anaconda3\envs\[conda_env]\lib\threading.py", line 946, in run
    self._target(*self._args, **self._kwargs)
  File "...\Anaconda3\envs\[conda_env]\lib\asyncio\base_events.py", line 600, in run_forever
    self._run_once()
  File "...\Anaconda3\envs\[conda_env]\lib\asyncio\base_events.py", line 1860, in _run_once
    event_list = self._selector.select(timeout)
  File "...\Anaconda3\envs\[conda_env]\lib\selectors.py", line 324, in select
    r, w, _ = self._select(self._readers, self._writers, [], timeout)
  File "...\Anaconda3\envs\[conda_env]\lib\selectors.py", line 315, in _select
    r, w, x = select.select(r, w, w, timeout)
ValueError: too many file descriptors in select()

> dvc version       
DVC version: 2.13.0 (pip)
---------------------------------
Platform: Python 3.10.4 on Windows-10-10.0.19044-SP0
Supports:
        azure (adlfs = 2022.7.0, knack = 0.9.0, azure-identity = 1.10.0),
        webhdfs (fsspec = 2022.5.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)

[efiop]

@friggog Have you tried lowering the --jobs, btw?

[friggog]

Yes! That fixes it 😃 - the default of 2x CPU cores seems a bit crazy on machines with many cores (I have 18 so get 36 threads). Would it make sense to cap the default at something sensible like 10-15?

[efiop]

@friggog It really depends on particular filesystem. Right now we are working on not using default jobs number at all and relying on the underlying fsspec filesystem's async functionality to throttle as needed (e.g. we use adlfs for azure). So please stay tuned for that.