Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add best practice how to store results #8

Closed
m-mohr opened this issue Nov 19, 2019 · 7 comments
Closed

Add best practice how to store results #8

m-mohr opened this issue Nov 19, 2019 · 7 comments
Assignees

Comments

@m-mohr
Copy link
Member

m-mohr commented Nov 19, 2019

Based on issue Open-EO/openeo-processes#59 and a recent discussion:

Currently, users have no knowledge of how many files they will receive (and what exactly will be in it/them) after using save_result, as the behavior of this process is left to the back-end.

Let's assume that the job output is composed by more than one raster, i.e. it is a datacube with dimensions (x,y,A,B, ...) where dimensions A, B, ... have more than one element (a typical example is a timeseries with multiple bands).
We agreed on the following:

  • the back-end freely decides how to save and return job outputs, but it MUST expose on /output_formats information on how datacubes with mutliple dimensions will be offered to the user (e.g. one file per timestamp per band; one file per timestamp with n bands in each, etc ...)
  • generally we define best-practice to save one file per timestamp (each file may contain multiple bands, or whichever other dimension)

Quote from @lforesta

@m-mohr m-mohr transferred this issue from Open-EO/openeo-api Jan 14, 2020
@m-mohr
Copy link
Member Author

m-mohr commented Aug 28, 2020

Is there still a desire to work on this @lforesta?

@lforesta
Copy link

lforesta commented Sep 1, 2020

Thanks for the reminder :)
'desire' is a big word but I think it would be useful if backends add this information for users. My idea would be to have an additional field at GET /file_formats under output, maybe called output-structure.
Depending how strict we want to be we can make this field non-nullable.

If the idea sounds fine I can put together a PR

@m-mohr
Copy link
Member Author

m-mohr commented Sep 1, 2020

API changes are bit harder to do nowadays. They can't be breaking and we won't release a new version in the openEO project runtime any more. So we only have the description field for now.

I think at some point we discussed to make this only a best practice (that's why it's likely an issue at openeo.org, no openeo-api). In this case we just need a write-up on how file should be stored/made available by back-ends. Or is this not required any longer as we have STAC now for results, which can transfer a lot of additional information per asset/file?

@lforesta
Copy link

lforesta commented Sep 2, 2020

Yes actually now this information may be provided by the backend in properties/description when sensing a request to GET /jobs/{job_id}/results, so no need to add that to openeo.org I think

@m-mohr
Copy link
Member Author

m-mohr commented Sep 2, 2020

And we may even extend STAC with more information, for example it can describe the bands and gsd per asset already and it is planned to add information such as datatype, min/max values etc. So I think we are fine with that approach. Even better :-)

@m-mohr m-mohr closed this as completed Sep 2, 2020
@clausmichele
Copy link
Member

After trying the sample process graph with climatological_normal, I've noticed that in the process graph there's save_result with the palette option, how do you actually apply this? I do not understand what is mapping with 0 -> #2166AC 1 -> #4393C3 . Is the value 0 in the result mapped to the color #2166AC? What to do with values which are not mapped with a color?

image

@m-mohr
Copy link
Member Author

m-mohr commented Oct 7, 2020

@clausmichele This is earthengine driver related and needs to be discussed there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants