BF: do not use BaseDownloader instance wide InterProcessLock - resolves stalling or errors during parallel installs

[yarikoptic]

apparently that does not play nice with multi-threading parallelization,
where (I guess) we instantiate and reuse the same Downloader across multiple
threads, and thus reusing the lock.

Bu instantiating the lock right before using it, albeit may be adding even more runtime
overhead, we are avoiding the problems with InterProcessLock.

Closes #6483 and also see harlowja/fasteners#91 for more
information. Filed also #6506 which observed while troubleshooting.

I am yet to try it on actual original HCP use case, but with this script
http://www.onerussian.com/tmp/ts-parallel-download.py
and this list of urls for our store http://www.onerussian.com/tmp/store-configs.txt
/bin/rm -f /tmp/out/*; head -n 1000 /tmp/store-configs.txt | python ts-parallel-download.py now works ok

edit: HCP timing on smaug:

action summary:
  install (ok: 4547)
                 datalad get -n -r HCP1200 --jobs 32  11158.83s user 7123.09s system 176% cpu 2:52:38.36 total

[adswa]

I was about to try it on the hcp dataset on our cluster, but it runs Python 3.7, where it won't use threads... :/

[yarikoptic]

I was about to try it on the hcp dataset on our cluster, but it runs Python 3.7, where it won't use threads... :/

can merge into master locally -- that one should be ok with 3.7 and threads.

[yarikoptic]

fails on macs are "known" (I think) due to newer annex. Otherwise seems to be green.

[yarikoptic]

ok, tested also on a time datalad install -J32 -r https://github.com/datalad-datasets/human-connectome-project-openaccess which completed in 3:41:08.86 total . no stalls/errors. Let's consider it "the right fix" and merge/release it (what to wait for?)

edit: I will also merge into master

[adswa]

I'm trying it locally, and it seems to run smoothly with --jobs 5. I saw a download locking thing with --jobs 10, but that may have been my computer being overwhelmed with processes.

[yarikoptic]

Thank you @adswa for testing! Locking with 10 sounds worrisome, could be some other issue though. Any interesting log messages were displayed?

[adswa]

Nothing that caught my eye, but it was with default info log level. I accidentally filled my hard drive just now, but once I have space again I can retry.

[yarikoptic]

I think looking at lsof and py-spy top were also useful for me to see where in datalad prices we might be at the moment, whenever it clearly wasn't done child annex process