Collections key not found with library

[stav]

I'm curious about the difference between Collection.get() and Collection.iter(key=[KEY])

>>> key = '456/789'
>>> store = project.collections.get_store('trump')
>>> store.set({'_key': key, 'value': 'abc'})
>>> print(store.list(key=[key]))

[{'value': 'abc', '_key': '456/789'}]  # https://storage.scrapinghub.com/collections/9328/s/trump?key=456%2F789&meta=_key

>>> try:
>>>     print(store.get(key))
>>> except scrapinghub.client.exceptions.NotFound as e:
>>>     print(getattr(e, 'http_error', e))

404 Client Error: Not Found for url: https://storage.scrapinghub.com/collections/9328/s/trump/456/789

I assume that Collection.get() is a handy shortcut for the key-filtered .iter() function so I guess the point of my issue is that .get() will raise an exception if given bad input, for example slashes

[vshlapakov]

As you can see by the URLs, the functions work a bit differently, i.e. using different API endpoints.

In the first case, when using .list(), it calls scan logic based on the given parameters (an item key), and all works fine. In the second case, when using .get(), it tries to get a single item from the collection with a different endpoint, and it fails because there's a slash character in the middle of the key, while it works fine in other cases. The endpoint route excludes a slash character in the key name, to avoid messing with other endpoints routes.

However, I don't see there are any restrictions on collection item key format as of now, so the input is actually correct, and I'd say it's the endpoint who doesn't handle it properly. Another option is to enforce some rules on collection item keys, but that might affect existing collections, so it should be done very carefully. I'll create an internal ticket to discuss and handle that in the future, thanks for the report!

[stav]

Good deal, thanks for the explanation

[ejulio]

I think we could add some docs here, or here https://doc.scrapinghub.com/api/collections.html#access-a-record about this subject. Last week it took me a few minutes to discover that everything was ok, but the endpoint did not allow for a / in the key name.

[stav]

https://github.com/scrapinghub/doc.scrapinghub.com/pull/187

[ejulio]

Given @stav PR above, I think we could work on a fix for .get() to always query using query string instead of pretty urls. What do you think?