Added clip_start_end_year preprocessor

[schlunma]

This PR adds an clip_start_end_year preprocessor which is necessary to move the extract_time, extract_month and extract_season preprocessors down in the preprocessor order which allows their use with custom_order: true.

Tasks

• Create an issue to discuss what you are going to do, if you haven't done so already (and add the link at the bottom)
• This pull request has a descriptive title that can be used in a changelog
• Add unit tests
• Public functions should have a numpy-style docstring so they appear properly in the API documentation. For all other functions a one line docstring is sufficient.
• If writing a new/modified preprocessor function, please update the documentation
• Circle/CI tests pass. Status can be seen below your pull request. If the tests are failing, click the link to find out why.
• Codacy code quality checks pass. Status can be seen below your pull request. If there is an error, click the link to find out why. If you suspect Codacy may be wrong, please ask by commenting.
• Please use yamllint to check that your YAML files do not contain mistakes
• If you make backward incompatible changes to the recipe format, make a new pull request in the ESMValTool repository and add the link below

If you need help with any of the tasks above, please do not hesitate to ask by commenting in the issue or pull request.

---

Closes #762

[bouweandela]

It would be good to take #345 into account, at least for the naming of the new preprocessor function.

[schlunma]

Ready for review

[valeriupredoi]

wait - am thoroughly confused - now we have extract_timerange which is a more specific case of extract_time and they are both preprocessor functions and only extract_timerange is used in the recipe for data filtering, that doesn't support filtering on months, which is needed in cases when we have data ending in weird months like, say, October - @schlunma can you pls tell me the logic behind this - I see both a duplicated functionality and the use of a less performant function at the same time (yeah, am cranky today 😁 ) 🍺 - also, have you changed the doumentation?

[valeriupredoi]

see me comment above 🍺

[schlunma]

Hey V., sorry I somehow didn't post my answer on your question above (which I already typed 😬).

So here is the thing:
extract_timerange should NOT be used by the user, it's only an internal function for extracting the start_year and end_year given by the recipe (similar to other "hidden" preprocessor functions like load or concatenate). That's the reason why we don't need documentation for it. extract_time can now be used by the user and behaves like a "regular" preprocessor now (i.e. it can be shuffled around with custom_order=true and it is not used by default anymore).

I really think that we should merge this soon, other people ran into this issue as well: #762 (comment) and ESMValGroup/ESMValTool#1864 (comment).

[valeriupredoi]

ok I got persuaded by @schlunma - looks like something more flexible and frontend-y than the internal backend function 👍

[bouweandela]

@bascrezee @lukasbrunner @Peter9192 This looks like something you need #796 (comment), would one of you be willing to try if it works for your use case?

[Peter9192]

Anyone currently reviewing this? Otherwise I can have a go at it

[schlunma]

Yes, please have a go at it @Peter9192. Thanks! ☺️

[Peter9192]

Code looks good @schlunma ! I left a few comments, will now proceed with some offline testing.
One general question: is extract_timerange really the most suitable name for this preprocessor, or can we come up with something slightly more distinguishing? clip_start_endyear is the best I could come up with so far.

[Peter9192]

Pytest locally ran fine after merging origin/master into this branch, and I tested with a custom recipe, which also produced the expected results. So I'm quite happy with the code. If you could just address my comments/questions above, then I think we're good to go.

test diagnostic

documentation:
  description: test PR
  authors:
    - kalverla_peter

preprocessors:
  tas_anomalies:
    custom_order: true
    area_statistics:
      operator: mean
    anomalies:
      period: full
      reference:
        start_year: 1981
        start_month: 1
        start_day: 1
        end_year: 2010
        end_month: 12
        end_day: 31
      standardize: false
    extract_time:
      start_year: 2081
      start_month: 1
      start_day: 1
      end_day: 31
      end_month: 12
      end_year: 2099

diagnostics:
  test1:
    variables:
      tas:
        start_year: 1965
        end_year: 2099
        mip: Amon
        preprocessor: tas_anomalies
        additional_datasets:
          - {dataset: ACCESS1-0, project: CMIP5, exp: [historical, rcp85], ensemble: r1i1p1}
    scripts:
      quickplot:
        script: examples/diagnostic.py
        quickplot:
          plot_type: plot

[schlunma]

Thanks @Peter9192 for your review! 👍 I included all your comments, so this PR should be ready now.

[Peter9192]

Thanks @schlunma I think you missed one rename, but otherwise I'm happy to see this merged