Add size_policy to HTTPFile

[martindurant]

Ref #45

[martindurant]

Thoughts on this, @rabernat ? There are quite a few things now that a user might need to know, which are pretty hard to explain, but flexibility that can be very useful. size_policy can be an arg to HTTPFileSystem (without the int option).

[rabernat]

Have you tested this branch on my url?

https://data.nas.nasa.gov/ecco/download_data.php?file=/eccodata/llc_4320/compressed/0000010368/Theta.0000010368.data.shrunk

[martindurant]

Yes, like

f_obj = fs.open(path, size_policy='get')

or give the exact file size, since you happen to know in this case.

[rabernat]

Then it solves my problem. Thanks so much for this quick fix!

[rabernat]

How would one hope to test for this?

Does the test suite actually spin up or mock an http server?

[martindurant]

Yes, it spins up a python standard http file server, which has specific behaviours of its own. To get a full set of tests for all the different things servers might do... would take some pretty sophisticated mocking. Maybe could be done better with various known-behaviour docker images.