Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MTG: get projection and extent information from file #845

Merged
merged 35 commits into from Apr 3, 2020

Commits on Nov 22, 2019

  1. MTG: get projection and extent information from file

    In the MTG FCI FDHSI L1C reader, get the projection coordinates and
    extent information from file.  For the projection coordinates, get those
    from /data/mtg_geos_projection rather than from /state/processor, as
    is consistent with the documentation.  The extent information was
    previously hardcoded but now obtained from data/channel/measured/x and
    data/channel/measured/y, which contain this information in radians.
    This commit fixes pytroll#840 although there are still some issues with the
    resulting image that need to be resolved.
    gerritholl committed Nov 22, 2019
    Copy the full SHA
    41a38c7 View commit details
    Browse the repository at this point in the history
  2. FCI: Use pixel outer corners rather than centres

    In FCI projection area extent, use the pixel outer corners rather than the centres.
    gerritholl committed Nov 22, 2019
    Copy the full SHA
    1015803 View commit details
    Browse the repository at this point in the history
  3. TST: Expand mocking data on FCI projection

    Newly the FCI reader is using new fields it wasn't using before.
    Make sure the test routines are mocking these fields so that the tests
    can run.
    gerritholl committed Nov 22, 2019
    Copy the full SHA
    556fde1 View commit details
    Browse the repository at this point in the history
  4. Replace f-string by regular string

    Since we still support Python 2, replace f-string by regular string in
    FCI L1C FDHSI reader
    gerritholl committed Nov 22, 2019
    Copy the full SHA
    569af22 View commit details
    Browse the repository at this point in the history
  5. Remove unused attribute from FCI FileHandler

    Remove the unused ``_fallback_area_def`` attribute from the FCI L1C
    FDHSI FileHandler class.
    gerritholl committed Nov 22, 2019
    Copy the full SHA
    372de7d View commit details
    Browse the repository at this point in the history
  6. DOC: Added info on how geolocation is done

    Added to the reader module docstring information on how the geolocation
    is performed.
    gerritholl committed Nov 22, 2019
    Copy the full SHA
    0739a40 View commit details
    Browse the repository at this point in the history
  7. Copy the full SHA
    7ff36f7 View commit details
    Browse the repository at this point in the history
  8. Disable netcdf load() workaround for xarray≥0.13

    Earlier this year, I programmed a workaround for a bug in xarray.  This
    bug got fixed in xarray 0.13.  This commit disabled the workaround for
    sufficiently recent versions of xarray.  Hopefully this will improve
    performance.
    gerritholl committed Nov 22, 2019
    Copy the full SHA
    6b3e373 View commit details
    Browse the repository at this point in the history

Commits on Nov 25, 2019

  1. FCI will use h5netcdf backend

    The FCI reader now uses the h5netcdf backend.  See the comments on issue
    972 for the rationale.  The netcdf4 backend has a memory leak and even
    worse performance than the h5netcdf backend for opening and closing
    files many times, which the filehandler does.
    
    This addresses pytroll#972 although I'm not yet fully satisfied.
    gerritholl committed Nov 25, 2019
    Copy the full SHA
    b4c0a90 View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2019

  1. FCI: improve attribute reading

    When reading attributes, do not access the data variable.
    Accessing the data variable will trigger netcdf_utils to
    actually open (and close) the netcdf file using xarray,
    causing unneccessary overhead.  Instead, use the fact that
    the netcdf reader has cached this information.
    
    More caching may be needed.
    
    This commit is related to pytroll#972.
    gerritholl committed Nov 26, 2019
    Copy the full SHA
    d7c5c84 View commit details
    Browse the repository at this point in the history

Commits on Nov 27, 2019

  1. FCI performance: cache area extents between channels

    For channels with the same resolution, don't recalculate the area extent
    as this is a performance hit.  Instead, cache the area extent between
    channels with the same resolution.
    gerritholl committed Nov 27, 2019
    Copy the full SHA
    e02944c View commit details
    Browse the repository at this point in the history
  2. FCI perf bugfix, don't share cache between instances

    Fix a bug in the FCI performance optimisation, when caching area extents
    do not share this cache between different instances.
    gerritholl committed Nov 27, 2019
    Copy the full SHA
    abea103 View commit details
    Browse the repository at this point in the history
  3. Copy the full SHA
    0957e94 View commit details
    Browse the repository at this point in the history
  4. Fix nlines/ncols bug in FCI reader

    FCI filehandler was setting nlines and ncols as an attribute, but this is
    channel dependent.  Store as a dictionary instead.  I don't know why
    this did not trigger any problems before the recent optimisations.
    gerritholl committed Nov 27, 2019
    Copy the full SHA
    b59a171 View commit details
    Browse the repository at this point in the history
  5. Bugfix, write ncols/nrows before returning cache

    Bugfix in fci reader, now that I'm sometimes returning stuff from cache,
    make sure the side-effects such as setting other attributes has been
    carried out already...
    gerritholl committed Nov 27, 2019
    Copy the full SHA
    6bfd479 View commit details
    Browse the repository at this point in the history
  6. FCI reader, don't needlessly store state

    Needlessly storing the state is just asking for problems, as I
    experienced.  Don't store ncols and nrows as attributes, instead pass
    them on between methods.  This is not a property of a filehandler.
    gerritholl committed Nov 27, 2019
    Copy the full SHA
    fcd0354 View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2019

  1. FCI, don't store area as attribute

    In the FCI reader, do not store the area definition as a first-level
    instance attribute.  This is different between channels.
    gerritholl committed Nov 28, 2019
    Copy the full SHA
    a244b7b View commit details
    Browse the repository at this point in the history
  2. FCI: move area caching one level up

    Instead of caching the area extent, cache the full area definition.
    gerritholl committed Nov 28, 2019
    Copy the full SHA
    5884cd2 View commit details
    Browse the repository at this point in the history

Commits on Nov 29, 2019

  1. FCI: do not duplicate changes from other branch

    Do not duplicate the performance improvements from the other branch.
    gerritholl committed Nov 29, 2019
    Copy the full SHA
    5be0e74 View commit details
    Browse the repository at this point in the history

Commits on Dec 3, 2019

  1. Copy the full SHA
    66cb17e View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2019

  1. Copy the full SHA
    a08c620 View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2019

  1. FCI, pass inverse flattening to AreaDefinition

    In the FCI reader area definition, pass the inverse flattening, as
    included in the file, from the reader to the `AreaDefinition` into
    pyresample.
    gerritholl committed Dec 12, 2019
    Copy the full SHA
    489d782 View commit details
    Browse the repository at this point in the history

Commits on Dec 19, 2019

  1. Add corrected scale factors for FCI projection

    The scale factors and add offsets for the MTG FCI test data as published
    in 2019-09 are factually incorrect.  This probably causes the
    differences between the pytroll and FCI geolocations.  Until this is
    fixed from EUMETSATs side, hardcode corrected add_offset and
    scale_factor values.
    
    Also correct for an off-by-one error in the scale factors.
    gerritholl committed Dec 19, 2019
    Copy the full SHA
    2fb5693 View commit details
    Browse the repository at this point in the history
  2. Add corrected scale factors for FCI projection

    The scale factors and add offsets for the MTG FCI test data as published
    in 2019-09 are factually incorrect.  This probably causes the
    differences between the pytroll and FCI geolocations.  Until this is
    fixed from EUMETSATs side, hardcode corrected add_offset and
    scale_factor values.
    
    Similarly, there was a small error in the semi minor axis.
    
    Also correct for an off-by-one error in the scale factors.
    gerritholl committed Dec 19, 2019
    Copy the full SHA
    212d2ca View commit details
    Browse the repository at this point in the history
  3. Undo hardcoding FCI semi-minor axis

    The FCI semi-minor axis correction does not improve anything, remove
    this hardcoding correction.
    gerritholl committed Dec 19, 2019
    Copy the full SHA
    997a953 View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2020

  1. Copy the full SHA
    4689c99 View commit details
    Browse the repository at this point in the history
  2. Copy the full SHA
    2e84f15 View commit details
    Browse the repository at this point in the history
  3. FCI reader flake8 fixes

    gerritholl committed Jan 14, 2020
    Copy the full SHA
    8c96730 View commit details
    Browse the repository at this point in the history
  4. FCI, complement mocking of variable attributes

    Expand the mocking of variable attributes for the routine testing the
    FCI reader.  One attribute was missing, and none were created in the
    "/attr/..." caching in which they are accessed by the reader.
    gerritholl committed Jan 14, 2020
    Copy the full SHA
    f537a14 View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2020

  1. Copy the full SHA
    9ddf4a1 View commit details
    Browse the repository at this point in the history

Commits on Apr 2, 2020

  1. FCI: Remove hardcoded parameters

    Remove the hardcoded scale factor, add offset, and sweep.  Those had
    been added to correct for errors in the 2019-09 release of FCI testdata,
    but should be fixed in the upcoming release, so should be no longer
    needed.
    gerritholl committed Apr 2, 2020
    Copy the full SHA
    fb0540e View commit details
    Browse the repository at this point in the history
  2. FCI: Remove unused hardcoded dictionary

    From the FCI reader, remove an unused hardcoded dictionary containing
    extent information.  This information was only used for debugging, but
    is redundant relative to what is available in the file, assuming a full
    disk image is square.
    gerritholl committed Apr 2, 2020
    Copy the full SHA
    e500e9d View commit details
    Browse the repository at this point in the history
  3. FCI: doc/comment fixes and change exception thrown

    In the FCI reader, make minor fixes to comments and docstring, and
    replace a RuntimeError by a more specific ValueError if an invalid
    calibration key is passed.
    gerritholl committed Apr 2, 2020
    Copy the full SHA
    a8ddb7d View commit details
    Browse the repository at this point in the history

Commits on Apr 3, 2020

  1. FCI: flake8-docstring fix

    Apparently, PEP 257 insists that the closing quotes for a single-line docstring must be on the same line as the rest of the docstring.  I will abide by what is apparently the community standard, although I will hereby note my (minority?) preference for the "closing quotes on next line" alternative.
    
    Co-Authored-By: Martin Raspaud <martin.raspaud@smhi.se>
    gerritholl and mraspaud committed Apr 3, 2020
    Copy the full SHA
    1f969b0 View commit details
    Browse the repository at this point in the history
  2. FCI: Remove empty line after method docstring

    Remove an empty line immediately after a method docstring, in order to
    satisfy the flake8-docstrings plugin.
    gerritholl committed Apr 3, 2020
    Copy the full SHA
    c11926b View commit details
    Browse the repository at this point in the history