Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

quantiles support in aggregate_spatial #714

Open
soxofaan opened this issue Mar 7, 2024 · 1 comment
Open

quantiles support in aggregate_spatial #714

soxofaan opened this issue Mar 7, 2024 · 1 comment
Labels

Comments

@soxofaan
Copy link
Member

soxofaan commented Mar 7, 2024

Reported by @przell in https://discuss.eodc.eu/t/how-to-use-quantiles-on-a-2d-data-cube-x-y/703: aggregate_spatial does not seem to support quantiles process

reproduction example:

cube = con.load_collection(
    "SENTINEL2_L2A",
    temporal_extent=["2023-09-01", "2023-09-15"],
    bands=["B02"]
)
aggregates = cube.aggregate_spatial(
    geometries=shapely.geometry.box(3, 50, 3.001, 50.001),
    reducer=lambda data: data.quantiles([0.5])[0]
)
aggregates.download("forum703-aggregates.json")

gives

OpenEoApiError: [400] BadRequest: java.lang.IllegalArgumentException: The process Constants are not expected in a reducer callback, got value 0.5. is not supported to be used in a reducer callback. (ref: r-24030792f2cb445bbb8b9a8881b5f164)

@soxofaan
Copy link
Member Author

soxofaan commented Mar 7, 2024

full process graph for completeness:

{
  "process_graph": {
    "loadcollection1": {
      "process_id": "load_collection",
      "arguments": {
        "bands": ["B02"],
        "id": "SENTINEL2_L2A",
        "spatial_extent": null,
        "temporal_extent": ["2023-09-01", "2023-09-15"]
      }
    },
    "aggregatespatial1": {
      "process_id": "aggregate_spatial",
      "arguments": {
        "data": {"from_node": "loadcollection1"},
        "geometries": {
          "type": "Polygon",
          "coordinates": [
            [[3.001, 50], [3.001, 50.001], [3, 50.001], [3, 50], [3.001, 50]]
          ]
        },
        "reducer": {
          "process_graph": {
            "quantiles1": {
              "process_id": "quantiles",
              "arguments": {
                "data": {"from_parameter": "data"},
                "probabilities": [0.5]
              }
            },
            "arrayelement1": {
              "process_id": "array_element",
              "arguments": {"data": {"from_node": "quantiles1"}, "index": 0},
              "result": true
            }
          }
        }
      },
      "result": true
    }
  }
}

and stack trace in logs:

  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1594, in apply_process
    return process_function(args=ProcessArgs(args, process_id=process_id), env=env)
  File "/opt/venv/lib64/python3.8/site-packages/openeo_driver/ProcessGraphDeserializer.py", line 1142, in aggregate_spatial
    return cube.aggregate_spatial(geometries=geoms, reducer=reduce_pg, target_dimension=target_dimension)
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/geopysparkdatacube.py", line 1354, in aggregate_spatial
    visitor = GeotrellisTileProcessGraphVisitor(_builder=get_jvm().org.openeo.geotrellis.aggregate_polygon.SparkAggregateScriptBuilder()).accept_process_graph(reducer)
  File "/opt/venv/lib64/python3.8/site-packages/openeo/internal/process_graph_visitor.py", line 77, in accept_process_graph
    self.accept_node(graph[top_level_node])
  File "/opt/venv/lib64/python3.8/site-packages/openeo/internal/process_graph_visitor.py", line 88, in accept_node
    self._accept_process(process_id=pid, arguments=arguments, namespace=namespace)
  File "/opt/venv/lib64/python3.8/site-packages/openeo/internal/process_graph_visitor.py", line 100, in _accept_process
    self._accept_argument_dict(value)
  File "/opt/venv/lib64/python3.8/site-packages/openeo/internal/process_graph_visitor.py", line 118, in _accept_argument_dict
    self.accept_node(value['node'])
  File "/opt/venv/lib64/python3.8/site-packages/openeo/internal/process_graph_visitor.py", line 88, in accept_node
    self._accept_process(process_id=pid, arguments=arguments, namespace=namespace)
  File "/opt/venv/lib64/python3.8/site-packages/openeo/internal/process_graph_visitor.py", line 96, in _accept_process
    self._accept_argument_list(value)
  File "/opt/venv/lib64/python3.8/site-packages/openeo/internal/process_graph_visitor.py", line 113, in _accept_argument_list
    self.constantArrayElement(element)
  File "/opt/venv/lib64/python3.8/site-packages/openeogeotrellis/processgraphvisiting.py", line 65, in constantArrayElement
    self.builder.constantArrayElement(value)
  File "/usr/local/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__
    return_value = get_return_value(
  File "/usr/local/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o517246.constantArrayElement.
: java.lang.IllegalArgumentException: The process Constants are not expected in a reducer callback, got value 0.5. is not supported to be used in a reducer callback.
	at org.openeo.geotrellis.aggregate_polygon.SparkAggregateScriptBuilder.badReducer(SparkAggregateScriptBuilder.scala:92)
	at org.openeo.geotrellis.aggregate_polygon.SparkAggregateScriptBuilder.constantArrayElement(SparkAggregateScriptBuilder.scala:79)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)

@soxofaan soxofaan added the bug label Mar 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant