Scenario Failure: fusets_mogpr

[github-actions]

Benchmark Failure: fusets_mogpr

Scenario ID: fusets_mogpr
Backend System: openeofed.dataspace.copernicus.eu
Failure Count: 1
Timestamp: 2025-04-11 03:07:55

Links:

• Workflow Run: https://github.com/ESA-APEx/apex_algorithms/actions/runs/14394117429
• Scenario Definition: https://github.com/ESA-APEx/apex_algorithms/blob/91eb3e62cc9e6d179b655baa97132771b5b0c6e8/algorithm_catalog/vito/fusets_mogpr/benchmark_scenarios/fusets_mogpr.json
• Artifacts: https://github.com/ESA-APEx/apex_algorithms/actions/runs/14394117429#artifacts

---

Contact Information

Point of Contact:

Name	Organization	Contact
Bram Janssen	VITO	Contact via VITO (VITO Website, GitHub)

---

Process Graph

{
  "fusetsmogpr": {
    "process_id": "fusets_mogpr",
    "namespace": "https://raw.githubusercontent.com/ESA-APEx/apex_algorithms/09413be3c27e0e695d426c9ffe5a0fe90beefe65/openeo_udp/fusets_mogpr/fusets_mogpr.json",
    "arguments": {
      "s1_collection": "RVI",
      "s2_collection": "NDVI",
      "spatial_extent": {
        "coordinates": [
          [
            [
              5.178303838475193,
              51.252856237848164
            ],
            [
              5.178003609252369,
              51.25109194151486
            ],
            [
              5.179280940922463,
              51.25103833409551
            ],
            [
              5.179565949577788,
              51.25278555186941
            ],
            [
              5.178303838475193,
              51.252856237848164
            ]
          ]
        ],
        "type": "Polygon"
      },
      "temporal_extent": [
        "2021-01-01",
        "2021-12-15"
      ]
    }
  },
  "save1": {
    "process_id": "save_result",
    "arguments": {
      "data": {
        "from_node": "fusetsmogpr"
      },
      "format": "NETCDF"
    },
    "result": true
  }
}

---

Error Logs

scenario = BenchmarkScenario(id='fusets_mogpr', description='Multi output gaussian process regression example on NDVI timeseries'...14156285361!tests_test_benchmarks.py__test_run_benchmark_fusets_mogpr_!actual/job-results.json'}, reference_options={})
connection_factory = <function connection_factory.<locals>.get_connection at 0x7f7d8d3344a0>
tmp_path = PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_fusets_mogp0')
track_metric = <function track_metric.<locals>.append at 0x7f7d8d334860>
upload_assets_on_fail = <function upload_assets_on_fail.<locals>.collect at 0x7f7d8d334720>
request = <FixtureRequest for <Function test_run_benchmark[fusets_mogpr]>>

    @pytest.mark.parametrize(
        "scenario",
        [
            # Use scenario id as parameterization id to give nicer test names.
            pytest.param(uc, id=uc.id)
            for uc in get_benchmark_scenarios()
        ],
    )
    def test_run_benchmark(
        scenario: BenchmarkScenario,
        connection_factory,
        tmp_path: Path,
        track_metric,
        upload_assets_on_fail,
        request
    ):
        track_metric("scenario_id", scenario.id)
        # Check if a backend override has been provided via cli options.
        override_backend = request.config.getoption("--override-backend")
        backend = scenario.backend
        if override_backend:
            _log.info(f"Overriding backend URL with {override_backend!r}")
            backend = override_backend
    
        connection: openeo.Connection = connection_factory(url=backend)
    
        # TODO #14 scenario option to use synchronous instead of batch job mode?
        job = connection.create_job(
            process_graph=scenario.process_graph,
            title=f"APEx benchmark {scenario.id}",
            additional=scenario.job_options,
        )
        track_metric("job_id", job.job_id)
    
        # TODO: monitor timing and progress
        # TODO: abort excessively long batch jobs? https://github.com/Open-EO/openeo-python-client/issues/589
        job.start_and_wait()
    
        collect_metrics_from_job_metadata(job, track_metric=track_metric)
    
        results = job.get_results()
        collect_metrics_from_results_metadata(results, track_metric=track_metric)
    
        # Download actual results
        actual_dir = tmp_path / "actual"
        paths = results.download_files(target=actual_dir, include_stac_metadata=True)
        # Upload assets on failure
        upload_assets_on_fail(*paths)
    
        # Compare actual results with reference data
        reference_dir = download_reference_data(
            scenario=scenario, reference_dir=tmp_path / "reference"
        )
    
>       assert_job_results_allclose(
            actual=actual_dir,
            expected=reference_dir,
            tmp_path=tmp_path,
            rtol=scenario.reference_options.get("rtol", 1e-6),
            atol=scenario.reference_options.get("atol", 1e-6),
        )

tests/test_benchmarks.py:74: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

actual = PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_fusets_mogp0/actual')
expected = PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_fusets_mogp0/reference')

    def assert_job_results_allclose(
        actual: Union[BatchJob, JobResults, str, Path],
        expected: Union[BatchJob, JobResults, str, Path],
        *,
        rtol: float = _DEFAULT_RTOL,
        atol: float = _DEFAULT_ATOL,
        tmp_path: Optional[Path] = None,
    ):
        """
        Assert that two job results sets are equal (with tolerance).
    
        :param actual: actual job results, provided as :py:class:`~openeo.rest.job.BatchJob` object,
            :py:meth:`~openeo.rest.job.JobResults` object or path to directory with downloaded assets.
        :param expected: expected job results, provided as :py:class:`~openeo.rest.job.BatchJob` object,
            :py:meth:`~openeo.rest.job.JobResults` object or path to directory with downloaded assets.
        :param rtol: relative tolerance
        :param atol: absolute tolerance
        :param tmp_path: root temp path to download results if needed.
            It's recommended to pass pytest's `tmp_path` fixture here
        :raises AssertionError: if not equal within the given tolerance
    
        .. versionadded:: 0.31.0
    
        .. warning::
            This function is experimental and subject to change.
        """
        issues = _compare_job_results(actual, expected, rtol=rtol, atol=atol, tmp_path=tmp_path)
        if issues:
>           raise AssertionError("\n".join(issues))
E           AssertionError: Issues for file 'openEO.nc':
E           Issues for variable 'NDVI':
E           Left and right DataArray objects are not close
E           Differing values:
E           L
E               array([[[0.556007, 0.591284, ..., 0.736255, 0.789329],
E                       [0.525328, 0.508553, ..., 0.791798, 0.780141],
E                       ...,
E                       [0.684107, 0.71703 , ..., 0.700012, 0.645337],
E                       [0.769488, 0.678789, ..., 0.686924, 0.644514]],
E               
E                      [[0.530424, 0.58803 , ..., 0.7335  , 0.786636],
E                       [0.514004, 0.499301, ..., 0.774027, 0.768261],
E                       ...,
E                       [0.67218 , 0.81942 , ..., 0.698144, 0.641986],
E                       [0.811471, 0.699266, ..., 0.717683, 0.678197]],
E               
E                      ...,
E               
E                      [[0.548442, 0.582803, ..., 0.679925, 0.676125],
E                       [0.619265, 0.650734, ..., 0.634589, 0.658109],
E                       ...,
E                       [0.603054, 0.776919, ..., 0.652416, 0.566231],
E                       [0.750174, 0.74333 , ..., 0.700707, 0.697604]],
E               
E                      [[0.590392, 0.580427, ..., 0.679896, 0.710612],
E                       [0.619255, 0.645687, ..., 0.655338, 0.694114],
E                       ...,
E                       [0.617722, 0.746992, ..., 0.6511  , 0.565227],
E                       [0.702582, 0.733417, ..., 0.672428, 0.671679]]], shape=(236, 22, 13))
E           R
E               array([[[0.550219, 0.577186, ..., 0.739188, 0.79513 ],
E                       [0.524991, 0.502728, ..., 0.790443, 0.772449],
E                       ...,
E                       [0.685394, 0.717065, ..., 0.712917, 0.64336 ],
E                       [0.769518, 0.746715, ..., 0.657801, 0.64453 ]],
E               
E                      [[0.518626, 0.573838, ..., 0.736441, 0.792288],
E                       [0.512959, 0.4887  , ..., 0.766907, 0.761251],
E                       ...,
E                       [0.673456, 0.819515, ..., 0.710454, 0.640029],
E                       [0.811508, 0.781385, ..., 0.666755, 0.678191]],
E               
E                      ...,
E               
E                      [[0.549574, 0.579224, ..., 0.679155, 0.676029],
E                       [0.619176, 0.653985, ..., 0.629277, 0.657107],
E                       ...,
E                       [0.603202, 0.776989, ..., 0.656695, 0.566529],
E                       [0.7502  , 0.764151, ..., 0.637501, 0.697585]],
E               
E                      [[0.592416, 0.573211, ..., 0.679179, 0.709958],
E                       [0.61934 , 0.646251, ..., 0.65827 , 0.689235],
E                       ...,
E                       [0.617925, 0.747044, ..., 0.656705, 0.565497],
E                       [0.7026  , 0.753615, ..., 0.627991, 0.671678]]], shape=(236, 22, 13))
E           Issues for variable 'RVI':
E           Left and right DataArray objects are not close
E           Differing values:
E           L
E               array([[[0.614548, 0.447905, ..., 0.457153, 0.489423],
E                       [0.47748 , 0.460002, ..., 0.428476, 0.438832],
E                       ...,
E                       [0.348399, 0.454293, ..., 0.396434, 0.418475],
E                       [0.341637, 0.35157 , ..., 0.407204, 0.397936]],
E               
E                      [[0.504778, 0.448668, ..., 0.458794, 0.488725],
E                       [0.449876, 0.44329 , ..., 0.410042, 0.4288  ],
E                       ...,
E                       [0.288148, 0.321445, ..., 0.394268, 0.419207],
E                       [0.292775, 0.329841, ..., 0.44557 , 0.428153]],
E               
E                      ...,
E               
E                      [[0.391082, 0.449758, ..., 0.490712, 0.478931],
E                       [0.402198, 0.448477, ..., 0.468937, 0.479789],
E                       ...,
E                       [0.476982, 0.376589, ..., 0.414534, 0.4358  ],
E                       [0.364117, 0.346695, ..., 0.424396, 0.445564]],
E               
E                      [[0.617436, 0.450321, ..., 0.49073 , 0.533846],
E                       [0.430613, 0.446932, ..., 0.478615, 0.511272],
E                       ...,
E                       [0.554815, 0.415418, ..., 0.418135, 0.43602 ],
E                       [0.419508, 0.352887, ..., 0.389124, 0.422306]]], shape=(236, 22, 13))
E           R
E               array([[[0.582853, 0.451067, ..., 0.454976, 0.484119],
E                       [0.478964, 0.466471, ..., 0.438141, 0.451871],
E                       ...,
E                       [0.348399, 0.451718, ..., 0.416678, 0.419457],
E                       [0.342365, 0.382625, ..., 0.398838, 0.381893]],
E               
E                      [[0.477137, 0.45185 , ..., 0.456607, 0.482919],
E                       [0.44793 , 0.429918, ..., 0.400246, 0.43311 ],
E                       ...,
E                       [0.288148, 0.324534, ..., 0.41768 , 0.420194],
E                       [0.295649, 0.30866 , ..., 0.414422, 0.43156 ]],
E               
E                      ...,
E               
E                      [[0.404916, 0.450274, ..., 0.490618, 0.477386],
E                       [0.401082, 0.466401, ..., 0.471019, 0.484245],
E                       ...,
E                       [0.476982, 0.377327, ..., 0.439547, 0.436482],
E                       [0.363857, 0.345427, ..., 0.419262, 0.460176]],
E               
E                      [[0.583773, 0.452409, ..., 0.490604, 0.529693],
E                       [0.431105, 0.454984, ..., 0.499064, 0.541825],
E                       ...,
E                       [0.554815, 0.414501, ..., 0.439543, 0.436711],
E                       [0.416814, 0.367904, ..., 0.404322, 0.421949]]], shape=(236, 22, 13))

/opt/hostedtoolcache/Python/3.12.9/x64/lib/python3.12/site-packages/openeo/testing/results.py:386: AssertionError
----------------------------- Captured stdout call -----------------------------
0:00:00 Job 'cdse-j-250411021520476db9e85c85f6a8423f': send 'start'
0:00:13 Job 'cdse-j-250411021520476db9e85c85f6a8423f': created (progress 0%)
0:00:18 Job 'cdse-j-250411021520476db9e85c85f6a8423f': created (progress 0%)
0:00:25 Job 'cdse-j-250411021520476db9e85c85f6a8423f': created (progress 0%)
0:00:33 Job 'cdse-j-250411021520476db9e85c85f6a8423f': created (progress 0%)
0:00:43 Job 'cdse-j-250411021520476db9e85c85f6a8423f': created (progress 0%)
0:00:56 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:01:12 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:01:31 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:01:56 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:02:26 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:03:03 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:03:50 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:04:49 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:05:49 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:06:50 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:07:50 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:08:51 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:09:51 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:10:52 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:11:54 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:12:55 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:13:55 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:14:55 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:15:56 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:16:57 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:17:57 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:18:58 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:19:58 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:20:59 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:22:00 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:23:01 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:24:01 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:25:02 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:26:03 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:27:03 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:28:04 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:29:04 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:30:05 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:31:06 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:32:07 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:33:08 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:34:08 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:35:09 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:36:10 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:37:10 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:38:10 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:39:11 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:40:11 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:41:12 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:42:12 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:43:14 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:44:14 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:45:15 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:46:15 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:47:15 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:48:16 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:49:16 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:50:16 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:51:17 Job 'cdse-j-250411021520476db9e85c85f6a8423f': running (progress N/A)
0:52:18 Job 'cdse-j-250411021520476db9e85c85f6a8423f': finished (progress 100%)
------------------------------ Captured log call -------------------------------
INFO     conftest:conftest.py:110 Connecting to 'openeofed.dataspace.copernicus.eu'
INFO     openeo.config:config.py:193 Loaded openEO client config from sources: []
INFO     conftest:conftest.py:123 Checking for auth_env_var='OPENEO_AUTH_CLIENT_CREDENTIALS_CDSEFED' to drive auth against url='openeofed.dataspace.copernicus.eu'.
INFO     conftest:conftest.py:127 Extracted provider_id='CDSE' client_id='openeo-apex-benchmarks-service-account' from auth_env_var='OPENEO_AUTH_CLIENT_CREDENTIALS_CDSEFED'
INFO     openeo.rest.connection:connection.py:231 Found OIDC providers: ['CDSE']
INFO     openeo.rest.auth.oidc:oidc.py:401 Doing 'client_credentials' token request 'https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token' with post data fields ['grant_type', 'client_id', 'client_secret', 'scope'] (client_id 'openeo-apex-benchmarks-service-account')
INFO     openeo.rest.connection:connection.py:328 Obtained tokens: ['access_token', 'id_token']
INFO     openeo.rest.auth.oidc:oidc.py:401 Doing 'client_credentials' token request 'https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token' with post data fields ['grant_type', 'client_id', 'client_secret', 'scope'] (client_id 'openeo-apex-benchmarks-service-account')
INFO     openeo.rest.connection:connection.py:328 Obtained tokens: ['access_token', 'id_token']
INFO     openeo.rest.connection:connection.py:676 OIDC access token expired (403 TokenInvalid). Obtained new access token (grant 'client_credentials').
INFO     openeo.rest.auth.oidc:oidc.py:401 Doing 'client_credentials' token request 'https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token' with post data fields ['grant_type', 'client_id', 'client_secret', 'scope'] (client_id 'openeo-apex-benchmarks-service-account')
INFO     openeo.rest.connection:connection.py:328 Obtained tokens: ['access_token', 'id_token']
INFO     openeo.rest.connection:connection.py:676 OIDC access token expired (403 TokenInvalid). Obtained new access token (grant 'client_credentials').
INFO     openeo.rest.auth.oidc:oidc.py:401 Doing 'client_credentials' token request 'https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token' with post data fields ['grant_type', 'client_id', 'client_secret', 'scope'] (client_id 'openeo-apex-benchmarks-service-account')
INFO     openeo.rest.connection:connection.py:328 Obtained tokens: ['access_token', 'id_token']
INFO     openeo.rest.connection:connection.py:676 OIDC access token expired (403 TokenInvalid). Obtained new access token (grant 'client_credentials').
INFO     openeo.rest.auth.oidc:oidc.py:401 Doing 'client_credentials' token request 'https://identity.dataspace.copernicus.eu/auth/realms/CDSE/protocol/openid-connect/token' with post data fields ['grant_type', 'client_id', 'client_secret', 'scope'] (client_id 'openeo-apex-benchmarks-service-account')
INFO     openeo.rest.connection:connection.py:328 Obtained tokens: ['access_token', 'id_token']
INFO     openeo.rest.connection:connection.py:676 OIDC access token expired (403 TokenInvalid). Obtained new access token (grant 'client_credentials').
INFO     openeo.rest.job:job.py:404 Downloading Job result asset 'openEO.nc' from https://openeo.dataspace.copernicus.eu/openeo/1.1/jobs/j-250411021520476db9e85c85f6a8423f/results/assets/NmE3N2ZjZDEtOWMwOC00NmU5LWI4NzUtNTRmYjk5OWFiMjAw/57401a20e6e3e5054a06f496b813842e/openEO.nc?expires=1744945659 to /home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_fusets_mogp0/actual/openEO.nc
INFO     apex_algorithm_qa_tools.scenarios:util.py:341 Downloading reference data for scenario.id='fusets_mogpr' to reference_dir=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_fusets_mogp0/reference'): start 2025-04-11 03:07:45.260547
INFO     apex_algorithm_qa_tools.scenarios:util.py:341 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-14156285361!tests_test_benchmarks.py__test_run_benchmark_fusets_mogpr_!actual/openEO.nc' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_fusets_mogp0/reference/openEO.nc'): start 2025-04-11 03:07:45.260851
INFO     apex_algorithm_qa_tools.scenarios:util.py:347 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-14156285361!tests_test_benchmarks.py__test_run_benchmark_fusets_mogpr_!actual/openEO.nc' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_fusets_mogp0/reference/openEO.nc'): end 2025-04-11 03:07:47.209519, elapsed 0:00:01.948668
INFO     apex_algorithm_qa_tools.scenarios:util.py:341 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-14156285361!tests_test_benchmarks.py__test_run_benchmark_fusets_mogpr_!actual/job-results.json' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_fusets_mogp0/reference/job-results.json'): start 2025-04-11 03:07:47.209826
INFO     apex_algorithm_qa_tools.scenarios:util.py:347 Downloading source='https://s3.waw3-1.cloudferro.com/apex-benchmarks/gh-14156285361!tests_test_benchmarks.py__test_run_benchmark_fusets_mogpr_!actual/job-results.json' to path=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_fusets_mogp0/reference/job-results.json'): end 2025-04-11 03:07:48.512942, elapsed 0:00:01.303116
INFO     apex_algorithm_qa_tools.scenarios:util.py:347 Downloading reference data for scenario.id='fusets_mogpr' to reference_dir=PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_fusets_mogp0/reference'): end 2025-04-11 03:07:48.513128, elapsed 0:00:03.252581
INFO     openeo.testing.results:results.py:298 Comparing job results: PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_fusets_mogp0/actual') vs PosixPath('/home/runner/work/apex_algorithms/apex_algorithms/qa/benchmarks/tmp_path_root/test_run_benchmark_fusets_mogp0/reference')

[jdries]

@dsamaey : I suggest to set tolerance

[soxofaan]

I now force-ran the benchmark in https://github.com/ESA-APEx/apex_algorithms/actions/runs/14904038093/job/41862242429 with the more advanced diff reporting (using this trick),
but it failed before going into that code path because of earlier failures:

FAILED tests/test_benchmarks.py::test_run_benchmark[fusets_mogpr] - AssertionError: Issues for metadata file 'job-results.json':
Differing 'derived_from' links (110 common, 0 only in actual, 282 only in expected):
  only in actual: set()
  only in expected: {'/eodata/Sentinel-2/MSI/L2A_N0500/2021/01/07/S2B_MSIL2A_20210107T104329_N0500_R008_T31UFS_20230225T064751.SAFE', '/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2021/10/09/S1B_IW_GRDH_1SDV_20211009T173238_20211009T173303_029062_0377C7_13C4_COG.SAFE', '/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2021/02/15/S1B_IW_GRDH_1SDV_20210215T054956_20210215T055021_025613_030D69_04A4_COG.SAFE', '/eodata/Sentinel-2/MSI/L2A_N0500/2021/04/22/S2A_MSIL2A_20210422T104021_N0500_R008_T31UFS_20230520T072117.SAFE', '/eodata/Sentinel-2/MSI/L2A_N0500/2021/09/29/S2A_MSIL2A_20210929T104031_N0500_R008_T31UFS_20230126T060739.SAFE', '/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2021/10/12/S1A_IW_GRDH_1SDV_20211012T055839_20211012T055904_040082_04BEED_ED47_COG.SAFE', '/eodata/Sentinel-2/MSI/L2A_N0500/2021/01/15/S2A_MSIL2A_20210115T105411_N0500_R051_T31UFS_20230527T121116.SAFE', '/eodata/Sentinel-1/SAR/IW_GRDH_1S-COG/2021/08/25/S1A_IW_GRDH_1SDV_20210825T055838_20210825T055903_039382_04A6D7_4C9B_COG.SAFE', '/eodata/Sentinel-1/SAR/IW_G....
Issues for file 'openEO.nc':
Issues for variable 'NDVI':
Coordinates mismatch for dimension 't': ['2021-05-01T00:00:00.000000000' '2021-05-03T00:00:00.000000000'
 '2021-05-04T00:00:00.000000000' '2021-05-06T00:00:00.000000000'
 '2021-05-07T00:00:00.000000000' '2021-05-09T00:00:00.000000000'
...
'2021-08-10T00:00:00.000000000' '2021-08-11T00:00:00.000000000'
 '2021-08-13T00:00:00.000000000' '2021-08-14T00:00:00.000000000'] != ['2021-01-01T00:00:00.000000000' '2021-01-03T00:00:00.000000000'
 '2021-01-04T00:00:00.000000000' '2021-01-06T00:00:00.000000000'
 '2021-01-07T00:00:00.000000000' '2021-01-09T00:00:00.000000000'
 '2021-01-10T00:00:00.000000000' '2021-01-12T00:00:00.000000000'
...
'2021-12-06T00:00:00.000000000' '2021-12-08T00:00:00.000000000'
 '2021-12-09T00:00:00.000000000' '2021-12-11T00:00:00.000000000'
 '2021-12-12T00:00:00.000000000' '2021-12-14T00:00:00.000000000']
Shape mismatch: (70, 22, 13) != (236, 22, 13)
Issues for variable 'RVI':
Coordinates mismatch for dimension 't': ['2021-05-01T00:00:00.000000000' '2021-05-03T00:00:00.000000000'
 '2021-05-04T00:00:00.000000000' '2021-05-06T00:00:00.000000000'
 '2021-05-07T00:00:00.000000000' '2021-05-09T00:00:00.000000000'
 '2021-05-10T00:00:00.000000000' '2021-05-12T00:00:00.000000000'
...
'2021-08-04T00:00:00.000000000' '2021-08-05T00:00:00.000000000'
 '2021-08-07T00:00:00.000000000' '2021-08-08T00:00:00.000000000'
 '2021-08-10T00:00:00.000000000' '2021-08-11T00:00:00.000000000'
 '2021-08-13T00:00:00.000000000' '2021-08-14T00:00:00.000000000'] != ['2021-01-01T00:00:00.000000000' '2021-01-03T00:00:00.000000000'
 '2021-01-04T00:00:00.000000000' '2021-01-06T00:00:00.000000000'
 '2021-01-07T00:00:00.000000000' '2021-01-09T00:00:00.000000000'
 '2021-01-10T00:00:00.000000000' '2021-01-12T00:00:00.000000000'
...
'2021-12-06T00:00:00.000000000' '2021-12-08T00:00:00.000000000'
 '2021-12-09T00:00:00.000000000' '2021-12-11T00:00:00.000000000'
 '2021-12-12T00:00:00.000000000' '2021-12-14T00:00:00.000000000']
Shape mismatch: (70, 22, 13) != (236, 22, 13)
=========== 1 failed, 11 deselected, 3 warnings in 976.39s (0:16:16) ===========

[soxofaan]

change of time dimension is probably because of c3868cc

[soxofaan]

I compared the time series of the reference and actual in a couple of points, and there seem to be some significant differences:

(notebook to reproduce this: https://gist.github.com/soxofaan/11381f650dd64e79bf6b8b3295317a28)

@jdries Is it acceptable to update the reference data to these actuals, or does this require more investigation?

[jdries]

makes sense, please just update ref files