Scenario Failure: worldcereal_crop_type #255
Description
Benchmark scenario ID: worldcereal_crop_type
Benchmark scenario definition: https://github.com/ESA-APEx/apex_algorithms/blob/1ebea5ccefb95e5c66d42cda6ad8dcef7d53e7fd/algorithm_catalog/vito/worldcereal_crop_type/benchmark_scenarios/worldcereal_crop_type.json
openEO backend: openeofed.dataspace.copernicus.eu
GitHub Actions workflow run: https://github.com/ESA-APEx/apex_algorithms/actions/runs/18313653546
Workflow artifacts: https://github.com/ESA-APEx/apex_algorithms/actions/runs/18313653546#artifacts
Test start: 2025-10-07 13:07:09.388381+00:00
Test duration: 0:02:20.146119
Test outcome: ❌ failed
Last successful test phase: create-job
Failure in test phase: run-job
Contact Information
| Name | Organization | Contact |
|---|---|---|
| Kristof Van Tricht | VITO | Contact via VITO (VITO Website, GitHub) |
Process Graph
{
"worldcerealcropextent1": {
"process_id": "worldcereal_crop_type",
"namespace": "https://raw.githubusercontent.com/WorldCereal/worldcereal-classification/refs/tags/worldcereal_crop_type_v1.0.0/src/worldcereal/udp/worldcereal_crop_type.json",
"arguments": {
"spatial_extent": {
"west": 622694.5968575787,
"east": 623079.000934101,
"south": 5672232.857114074,
"north": 5672519.995940826,
"crs": "EPSG:32631",
"srs": "EPSG:32631"
},
"temporal_extent": [
"2018-05-01",
"2019-04-30"
],
"model_url": "https://s3.waw3-1.cloudferro.com/swift/v1/APEx-benchmarks/worldcereal_crop_type/test_worldcereal_crop_type_custommodel.onnx"
},
"result": true
}
}Error Logs
scenario = BenchmarkScenario(id='worldcereal_crop_type', description='WorldCereal crop type benchmark', backend='openeofed.datasp...gorithms/apex_algorithms/algorithm_catalog/vito/worldcereal_crop_type/benchmark_scenarios/worldcereal_crop_type.json'))
connection_factory = <function connection_factory.<locals>.get_connection at 0x7f140bdcd800>
tmp_path = PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_worldcereal0')
track_metric = <function track_metric.<locals>.track at 0x7f140bdcdb20>
track_phase = <function track_phase.<locals>.track at 0x7f140bdcd940>
upload_assets_on_fail = <function upload_assets_on_fail.<locals>.collect at 0x7f140bdcdbc0>
request = <FixtureRequest for <Function test_run_benchmark[worldcereal_crop_type]>>
@pytest.mark.parametrize(
"scenario",
[
# Use scenario id as parameterization id to give nicer test names.
pytest.param(uc, id=uc.id)
for uc in get_benchmark_scenarios()
],
)
def test_run_benchmark(
scenario: BenchmarkScenario,
connection_factory,
tmp_path: Path,
track_metric,
track_phase,
upload_assets_on_fail,
request,
):
track_metric("scenario_id", scenario.id)
with track_phase(phase="connect"):
# Check if a backend override has been provided via cli options.
override_backend = request.config.getoption("--override-backend")
backend_filter = request.config.getoption("--backend-filter")
if backend_filter and not re.match(backend_filter, scenario.backend):
# TODO apply filter during scenario retrieval, but seems to be hard to retrieve cli param
pytest.skip(
f"skipping scenario {scenario.id} because backend {scenario.backend} does not match filter {backend_filter!r}"
)
backend = scenario.backend
if override_backend:
_log.info(f"Overriding backend URL with {override_backend!r}")
backend = override_backend
connection: openeo.Connection = connection_factory(url=backend)
with track_phase(phase="create-job"):
# TODO #14 scenario option to use synchronous instead of batch job mode?
job = connection.create_job(
process_graph=scenario.process_graph,
title=f"APEx benchmark {scenario.id}",
additional=scenario.job_options,
)
track_metric("job_id", job.job_id)
with track_phase(phase="run-job"):
# TODO: monitor timing and progress
# TODO: abort excessively long batch jobs? https://github.com/Open-EO/openeo-python-client/issues/589
> job.start_and_wait()
tests/test_benchmarks.py:69:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/openeo/rest/job.py:346: in start_and_wait
job_info = self.describe()
^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/openeo/rest/job.py:103: in describe
return self.connection.get(f"/jobs/{self.job_id}", expected_status=200).json()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/openeo/rest/_connection.py:204: in get
return self.request("get", path=path, params=params, stream=stream, auth=auth, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/openeo/rest/connection.py:687: in request
return _request()
^^^^^^^^^^
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/openeo/rest/connection.py:680: in _request
return super(Connection, self).request(
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/openeo/rest/_connection.py:141: in request
self._raise_api_error(resp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <Connection to 'https://openeofed.dataspace.copernicus.eu/openeo/1.2/' with OidcBearerAuth>
response = <Response [404]>
def _raise_api_error(self, response: requests.Response):
"""Convert API error response to Python exception"""
status_code = response.status_code
try:
info = response.json()
except Exception:
info = None
# Valid JSON object with "code" and "message" fields indicates a proper openEO API error.
if isinstance(info, dict):
error_code = info.get("code")
error_message = info.get("message")
if error_code and isinstance(error_code, str) and error_message and isinstance(error_message, str):
raise OpenEoApiError(
http_status_code=status_code,
code=error_code,
message=error_message,
id=info.get("id"),
url=info.get("url"),
)
# Failed to parse it as a compliant openEO API error: show body as-is in the exception.
text = response.text
error_message = None
_log.warning(f"Failed to parse API error response: [{status_code}] {text!r} (headers: {response.headers})")
# TODO: eliminate this VITO-backend specific error massaging?
if status_code == HTTP_502_BAD_GATEWAY and "Proxy Error" in text:
error_message = (
"Received 502 Proxy Error."
" This typically happens when a synchronous openEO processing request takes too long and is aborted."
" Consider using a batch job instead."
)
> raise OpenEoApiPlainError(message=text, http_status_code=status_code, error_message=error_message)
E openeo.rest.OpenEoApiPlainError: [404] <html>
E <head><title>404 Not Found</title></head>
E <body>
E <center><h1>404 Not Found</h1></center>
E <hr><center>nginx/1.20.1</center>
E </body>
E </html>
/opt/hostedtoolcache/Python/3.12.11/x64/lib/python3.12/site-packages/openeo/rest/_connection.py:184: OpenEoApiPlainError
----------------------------- Captured stdout call -----------------------------
0:00:00 Job 'terrascope-j-2510071308094f01ad07bdcba8ee7696': send 'start'
0:00:50 Job 'terrascope-j-2510071308094f01ad07bdcba8ee7696': Service availability error while polling job status: [503] OidcProviderUnavailable: OIDC Provider is unavailable (ref: r-2510071308344478b63e740874075a94)
------------------------------ Captured log call -------------------------------
INFO conftest:conftest.py:131 Connecting to 'openeofed.dataspace.copernicus.eu'
INFO openeo.config:config.py:193 Loaded openEO client config from sources: []
INFO conftest:conftest.py:144 Checking for auth_env_var='OPENEO_AUTH_CLIENT_CREDENTIALS_CDSEFED' to drive auth against url='openeofed.dataspace.copernicus.eu'.
INFO conftest:conftest.py:148 Extracted provider_id='CDSE' client_id='openeo-apex-benchmarks-service-account' from auth_env_var='OPENEO_AUTH_CLIENT_CREDENTIALS_CDSEFED'
INFO openeo.rest.connection:connection.py:255 Found OIDC providers: ['CDSE']
INFO openeo.rest.auth.oidc:oidc.py:404 Doing 'client_credentials' token request 'https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token' with post data fields ['grant_type', 'client_id', 'client_secret', 'scope'] (client_id 'openeo-apex-benchmarks-service-account')
INFO openeo.rest.connection:connection.py:352 Obtained tokens: ['access_token', 'id_token']
WARNING openeo.rest._connection:_connection.py:174 Failed to parse API error response: [404] '<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx/1.20.1</center>\r\n</body>\r\n</html>\r\n' (headers: {'Content-Length': '153', 'Content-Type': 'text/html', 'Date': 'Tue, 07 Oct 2025 13:09:29 GMT'})