Make DownloadManager downloaded/extracted paths accessible

[albertvillanova]

Make accessible the file paths downloaded/extracted by DownloadManager.

Close #1831.

The approach:

• I set these paths as DownloadManager attributes: these are DownloadManager's concerns
• To access to these from DatasetBuilder, I set the DownloadManager instance as DatasetBuilder attribute: object composition

[lhoestq]

Nice thanks :)
And thanks for the test as well. I didn't know about the monkeypatch fixture of pytest, I'm used to the unittest patch function but I like this one too

I left a comment about how to handle multiple calls to .download in a row

[lhoestq]

Sometimes users call .download several times.
This happens for example if you want to first download and read some kind of manifest that tells you which other files you need to download.

Maybe this could be a dict or a list that is updated each time .download is called ?

[albertvillanova]

Well seen! Thanks for pointing this out, @lhoestq: as it can be called several times, new results should be appended at each call. I am refactoring to take this into account.

[lhoestq]

same here

[lhoestq]

Nice thank you :)
So right now if I understand correctly the .downloaded_paths is a list of Union[str, list, dict] right ? I was thinking of having a more user-friendly structure like a flattended list of str, or maybe a dict url->path. What do you think ?

It's possible to have a flattened list of paths with

self.downloaded_paths.extend(flatten_nested(downloaded_path_or_paths))

And if you think a dict would be more appropriate feel free to take a look at the _record_sizes_checksums method which does something really similar (but storing a dict url -> checksum_dict)

[albertvillanova]

First I was thinking of the dict, which makes sense for .download, mapping URL to downloaded path. However does this make sense for .extract, mapping the downloaded path to the extracted path? I ask this because the user did not chose the downloaded path, so this is completely unknown for them...

[lhoestq]

There could be several situations:

• download a file with no extraction
• download a file and extract it
• download a file, extract it and then inside the output folder extract some more files
• extract a local file (for datasets with data that are manually downloaded for example)
• extract a local file, and then inside the output folder extract some more files

So I think it's ok to have downloaded_paths as a dict url -> downloaded_path and extracted_paths as a dict local_path -> extracted_path.

[albertvillanova]

OK. I am refactoring this. I have opened #1879, as an intermediate step...

[lhoestq]

Nice thank you !