Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

general processor improvements #85

Merged
merged 16 commits into from
May 24, 2023
Merged

general processor improvements #85

merged 16 commits into from
May 24, 2023

Conversation

johntruckenbrodt
Copy link
Member

@johntruckenbrodt johntruckenbrodt commented May 11, 2023

This improves on the general processing workflow stability:

  • [dem.mosaic] the progress bar has been removed, which otherwise clutters logfiles in batch jobs
  • a new configuration parameter snap_gpt_args enables parametrization of the underlying SNAP GPT call; this has been found particularly relevant in an HPC environment where SNAP may try to use more resources than ordered in the SLURM job; all functions of the S1_NRB.snap have been extended with an argument gpt_args accordingly
  • snap.process: a new internal flag utm_multi enables processing to a single UTM zone if set to False; this is only intended for debugging and experimentation purposes
  • preprocessed GRDs are no longer deleted so that they can be reused for buffering of neighboring scenes; a mechanism to delete these files once they are no longer needed is yet to be implemented (closes Keep pre-processed GRD files for reuse #75, new issue: delete preprocessed GRD products #86)
  • GRD buffering: the method has been improved to no longer buffer a fixed number of source pixels but rather a multiple of the target pixel spacing in meters; the former might be insufficient if the target spacing is lower than the source spacing, in particular in combination with the edge cleaning performed during snap.postprocess

further bug fixes:

@johntruckenbrodt
Copy link
Member Author

I have a hunch that with the latest changes (or even before that) the function nrb.format and the underlying nrb.get_datasets don't ensure that incomplete tiles are created. This needs to be checked before merging this PR. Will get to it next week. @maawoo, FYI.

Before a scene was just removed from the list if no output was found.
This could result in incomplete tiles.
Before incomplete tiles could be created at the ends of a data take group because all tiles overlapping with the scenes were considered.
Now a second scene search is performed with the preselected tiles as input geometry and an extended time range so that pre- and succeeding scenes are selected as well.
@johntruckenbrodt johntruckenbrodt merged commit 3a87b3e into main May 24, 2023
@johntruckenbrodt johntruckenbrodt deleted the feature/snap_proc branch June 4, 2023 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Keep pre-processed GRD files for reuse
1 participant