Use SHUB_JOBAUTH environment variable in utils.parse_auth method

[victor-torres]

Currently, the parse_auth method tries to get the API Key from the SH_APIKEY environment variable, which needs to be manually set either in spider's code or in the [docker] image's code. A regular practice is to create dummy users and associate them with the project so that real contributors don't have to share their API Keys.

Another option is to use the credentials provided by the SHUB_JOBAUTH, defined during runtime when executing jobs in the Scrapy Cloud platform.

Although it's possible to use Collections and Frontera, this is not a regular Dash API Key but a JWT Token generated in runtime by JobQ service, which works only for a part of our API endpoints (JobQ/Hubstorage).

I'd like to contribute with a Pull Request adding support for this ephemeral API Key.

[victor-torres]

@vshlapakov @dangra @kalessin

[kalessin]

I personally like that feature. In the past that token couldn't be used to access frontier or collections, but seems it is possible now.

[dangra]

I personally like that feature. In the past that token couldn't be used to access frontier or collections, but seems it is possible now.

Not sure when this happened though. I have no objections as long as the token works for all api endpoints where it is used if SHUB_JOBAUTH is in the environment.

[victor-torres]

I cannot guarantee if it will work for every endpoint provided by this library, but it seems reasonable to prioritize the previous authentication methods and use SHUB_JOBAUTH as a fallback when running in cloud environments. What do you think?

[kalessin]

@dangra If the token doesn't work for some endpoint, nothing changes for the user that need to use it: it will just get an authentication error because no explicit apikey was passed. Notice that the feature is to use SHUB_JOBAUTH only if no explicit apikey was passed.

[vshlapakov]

It won't work for all API endpoints, that's for sure. But I like the idea of the fallback: if SH_APIKEY is not provided, check for SHUB_JOBAUTH and if there's a valid JWT token - use it with a warning describing its limitations and what should be done to overcome it.

[victor-torres]

I have just submitted a Pull Request with that suggestion (#104)

[vshlapakov]

Merged, thanks!