Paging failure from invalid `next` link returned from item search

[fmigneault]

Context

If a server returns an incorrect next link from limit/paging on item search, the iterator fails.

Should pystac-client be more robust toward this erroneous situation? Given that it knows the correct url initially (it should have succeeded once to initiate paging), and that another location would not make sense (assuming it is a "compliant" STAC API), this could be handled/patched transparently.

Note that https://www.eodms-sgdot.nrcan-rncan.gc.ca/stac/ (example below) is implemented using https://github.com/stac-utils/stac-fastapi (maybe a derived/old version?). I'm not sure if the problem persists upstream in stac-fastapi or not, as CRIM's own implementation patched paging on our end (crim-ca/stac-app#26).

Until reference servers fixes/updates their code, the pystac-client cannot be full leveraged against them even though it could patch it silently (though admittedly servers should also be fixed).

Example

Using POST https://www.eodms-sgdot.nrcan-rncan.gc.ca/stac/search with
(at the time of writing, it yields 116 items, therefore triggering paging)

{
  "limit": 100,
  "datetime": "2018-01-01T00:00:00Z/2025-05-20T23:59:59Z",
  "collections": [
    "rcm-ard"
  ],
  "intersects": {
    "type": "GeometryCollection",
    "geometries": [
      {
        "type": "Polygon",
        "coordinates": [[[-98.0, 50.25], [-98.0, 49.0], [-96.5, 49.0], [-96.5, 50.25], [-98.0, 50.25]]]
      }
    ]
  },
  "sortby": [
    {
      "field": "properties.datetime",
      "direction": "desc"
    }
  ]
}

Although url originally points at the correct location, its returned next link is invalid (missing /search).

Content of Link.extra_fields:

{'body': {'collections': ['rcm-ard'], 'datetime': '2018-01-01T00:00:00Z/2026-01-01T23:59:59Z', 'intersects': {'geometries': [{'coordinates': [[...]], 'type': 'Polygon'}], 'type': 'GeometryCollection'}, 'limit': 100, 'sortby': [{'direction': 'desc', 'field': 'properties.datetime'}], 'token': 'next:8bf30ecd-b61e-4b40-adb9-9cf027459ac7'}, 'href': 'https://www.eodms-sgdot.nrcan-rncan.gc.ca/stac/', 'method': 'POST', 'rel': 'next', 'type': 'application/geo+json'}

[gadomski]

We have a fair bit of code in pystac-client to handle irregular or invalid STAC APIs, so adding code to handle this case seems reasonable. I'd be happy to review a PR that does so.

I won't be implementing a fix myself (at least right away) because this seems much more a case of "fix your server" or, if it's not yours, "tell the folks that run the server to fix their server".

[fmigneault]

Agreed. I will try to talk with some contacts I have to apply fixes directly in the server.

For the PR, is there any strategy that should be considered? Eg: doing a try/except/retry around the self.read_json() call or some other approach?

[gadomski]

For the PR, is there any strategy that should be considered? Eg: doing a try/except/retry around the self.read_json() call or some other approach?

Yeah, I think catching an error, checking the url that was used for the paging request, and "correcting" it (with a warning) seems like a good approach. You'd want to make it configurable as well, so folks could turn on/off the behavior.

[jsignell]

This came up recently on #846 as well and it took me a while to figure out what was going on so I like the idea of at least making some effort to give an error message that points to this cause.

[fmigneault]

Discussed feature added in #852