feat(gcsfs/core): support storage_emulator_host for http requests

[awill1988]

Add support for specifying GCS emulator for this library.

I assume generating signed urls will not work, but for all other requests it should work fine.

While GCP does not release an official gcs emulator, there are a few community-driven projects:

• https://pypi.org/project/gcloud-storage-emulator/
• https://github.com/fsouza/fake-gcs-server

The official python library does have support for STORAGE_EMULATOR_HOST

• source code example

Fixes #424

[martindurant]

Can this be used to replace all of the vcrpy test code and recordings? Happy to use whichever emulator makes that possible.

[awill1988]

I'm not sure.. I don't have any experience with vcrpy. For my use case, I typically use a docker-compose stack to spin up those emulators. I have only used fake-gcs-emulator. Here's an example command for that:

docker run -p 4443:4443 fsouza/fake-gcs-server:latest -scheme http -public-host http://localhost:4443 -external-url http://localhost:4443

The reason for the change is because I'm leveraging this project in pyarrow and having the option lets me validate things locally. Really awesome project you guys have here!

Feel free to add commits or point me in the direction to update vcrpy tests.. I'd have to find some free time to look at migrating away from it to a emulation approach

[martindurant]

It would be super simple! Simply remove everything to do with vcrpy, and do exactly as you say, spin up a docker container as a pytest global fixture, and set the env variable to point at it. I can have a go at this myself next week, if you don't fancy it.

[awill1988]

I'll take a peek this weekend if I find a window of time. Otherwise I'll check-in next week. I'm good either way. At work I context switch a lot between Python, Go, Elixir so I don't feel the level of confidence I typically feel as prior to this job it was Typescript/JS. 😭 so many languages..

[martindurant]

I have started to move the tests to the emulator as opposed to vcrpy!
This will currently fail hard, as I've only done test_mapping, but so far so good.