Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

layers-to-h5 not working from bash #26

Closed
nickwg03 opened this issue Dec 16, 2019 · 8 comments
Closed

layers-to-h5 not working from bash #26

nickwg03 opened this issue Dec 16, 2019 · 8 comments
Labels
bug Something isn't working

Comments

@nickwg03
Copy link

Bug Description
When running rev-to-h5 from bash, the code appears to expect a key that is not previously set.

Full Traceback

Traceback (most recent call last):
  File "/home/ngrue/.conda-envs/rev-test/bin/reVX", line 11, in <module>
    load_entry_point('reVX', 'console_scripts', 'reVX')()
  File "/home/ngrue/.conda-envs/rev-test/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/ngrue/.conda-envs/rev-test/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/ngrue/.conda-envs/rev-test/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ngrue/.conda-envs/rev-test/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ngrue/.conda-envs/rev-test/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ngrue/.conda-envs/rev-test/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/ngrue/.conda-envs/rev-test/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/ngrue/reVX/reVX/cli.py", line 83, in layers_to_h5
    excl_h5 = ctx.obj['EXCL_H5']
KeyError: 'EXCL_H5'

Code Sample
Bash code:

reVX exclusions -h5 /projects/rev/data/exclusions/CONUS_Exclusions.h5 layers-to-h5 -l transfer.json

And transfer.json looks like this:

conus_interconnect.tif

Charge code
WEPG.10348.08.01.03
@nickwg03 nickwg03 added the bug Something isn't working label Dec 16, 2019
@nickwg03
Copy link
Author

@MRossol still hit a snag using /projects/rev/modulefiles environment. Looks like a dependency isn't being installed beforehand.

Traceback (most recent call last):
  File "/projects/rev/modulefiles/conda_env/bin/reVX", line 11, in <module>
    load_entry_point('reVX', 'console_scripts', 'reVX')()
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/cli.py", line 98, in layers_to_h5
    descriptions=descriptions)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/utilities/exclusions_converter.py", line 424, in layers_to_h5
    excls.geotiff_to_layer(layer, geotiff, description=description)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/utilities/exclusions_converter.py", line 378, in geotiff_to_layer
    chunks=self._chunks, description=description)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/utilities/exclusions_converter.py", line 297, in _geotiff_to_h5
    chunks=chunks)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/utilities/exclusions_converter.py", line 230, in _parse_tiff
    chunks=chunks)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/utilities/exclusions_converter.py", line 159, in _check_geotiff
    with Geotiff(geotiff, chunks=chunks) as tif:
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/handlers/geotiff.py", line 37, in __init__
    self._src = xr.open_rasterio(self._fpath, chunks=chunks)
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/xarray/backends/rasterio_.py", line 348, in open_rasterio
    from dask.base import tokenize
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/dask/base.py", line 12, in <module>
    from toolz import merge, groupby, curry, identity
ModuleNotFoundError: No module named 'toolz'

@MRossol
Copy link
Collaborator

MRossol commented Dec 16, 2019

I updated dask in the env, can you try again

@nickwg03
Copy link
Author

Still getting an odd error:.

The conus_interconnect.tif has the same shape as the arrays in CONUS_Exclusions.h5. It is in a different projection (starts out in ESRI:102008, but should readily convert to WGS84 like the rest of them).

I ran using the jupyter notebook and it worked. So this case isn't critical, but would be helpful to figure out for the future.

(/projects/rev/modulefiles/conda_env) [ngrue@r6i6n3 conus]$ reVX exclusions -h5 /projects/rev/data/exclusions/CONUS_Exclusions.h5 layers-to-h5 -l transfer.json
/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/pyproj/crs.py:77: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method.
  return _prepare_from_string(" ".join(pjargs))
Latitude coordinates conus_interconnect.tif and /projects/rev/data/exclusions/CONUS_Exclusions.h5 do not match!
Traceback (most recent call last):
  File "/projects/rev/modulefiles/conda_env/bin/reVX", line 11, in <module>
    load_entry_point('reVX', 'console_scripts', 'reVX')()
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/projects/rev/modulefiles/conda_env/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/cli.py", line 98, in layers_to_h5
    descriptions=descriptions)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/utilities/exclusions_converter.py", line 424, in layers_to_h5
    excls.geotiff_to_layer(layer, geotiff, description=description)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/utilities/exclusions_converter.py", line 378, in geotiff_to_layer
    chunks=self._chunks, description=description)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/utilities/exclusions_converter.py", line 297, in _geotiff_to_h5
    chunks=chunks)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/utilities/exclusions_converter.py", line 230, in _parse_tiff
    chunks=chunks)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/utilities/exclusions_converter.py", line 204, in _check_geotiff
    raise ExclusionsCheckError(error)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/utilities/exclusions_converter.py", line 204, in _check_geotiff
    raise ExclusionsCheckError(error)
  File "/lustre/eaglefs/projects/rev/modulefiles/gitrepos/reVX/reVX/utilities/exclusions_converter.py", line 198, in _check_geotiff
    raise ExclusionsCheckError(error)
reVX.utilities.exceptions.ExclusionsCheckError: Latitude coordinates conus_interconnect.tif and /projects/rev/data/exclusions/CONUS_Exclusions.h5 do not match!

@grantbuster
Copy link
Member

This is probably working as designed. The un projected coordinates probably vary a little and so the code is rejecting it. I can make a branch with a looser acceptance of differing coordinates and have a kwarg for required precision. Sound good?

@nickwg03
Copy link
Author

We'd presumably end up with lon/lats that are slightly different, right? Will that cause an issue during the KDTree operation, or would the longitude/latitude already loaded into the .h5 be used instead?

@grantbuster
Copy link
Member

The coordinates will not be overwritten, so the previous coordinates would be used in the techmap. In fact, if the techmap has already been established, the coordinates wont be used at all.

@nickwg03
Copy link
Author

yeah, then that sounds like it would work fine.

@grantbuster
Copy link
Member

Can you open a feature request and close this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@nickwg03 @grantbuster @MRossol