New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to open EPSG support file gcs.csv #673
Comments
@drnextgis are you getting this error with one of the linux wheels? Can you check one thing for me: whether or not you've set the GDAL_DATA and PROJ_LIB environment variables in your process? If you have, then Fiona's environment won't check in the wheel data for GDAL and PROJ support files. See https://github.com/Toblerity/Fiona/blob/master/fiona/_env.pyx#L211. |
Yes, I'm getting this error with linux wheel
but:
|
@sgillies could you reproduce this issue? |
I can reproduce this, using docker image |
@snorfalorpagus thanks for checking! I'm stuck on another project and no time to look at this today. |
Confirmed.
But
This is a different situation from the one in Rasterio. |
I had a moment of clarity this morning. The ensure_env_with_credentials decorator, when applied to fiona.open, is intended to make the following code
work like this
But it does not! Of course it does not. What actually happens is this:
There's no longer any GDAL environment when I'm certain we can get the behavior in the second block, though it may harder on Python 2.7. We will not be able to support this case:
except by setting GDAL_DATA in an environment variable or by decorating the crs getter. |
Ahhh good catch @sgillies! Do you like an idea of decorating CRS getter? |
Yes, I've almost got that working @drnextgis . We can't easily decorate class methods using I want to amend my comment above. The behavior in the 2nd block above will be rather difficult and in my judgement not worth doing. |
Done. |
I've got the same issue with the latest version of Fiona (1.8.1):
|
@drnextgis this sort of usage is not what I had in mind for Fiona or Rasterio and I don't want to change the code a lot to support it. I think the best option for you would be to set GDAL_DATA and PROJ_LIB environment variables. The problem, of course, is that Fiona and Rasterio don't have an API for getting these paths, you must know internal details, and that's not good. What would you think about the following proposal? When @snorfalorpagus I'd be grateful for a sanity check on this proposal from you, too. |
Sounds good to me and I see you already implemented that suggestion. @sgillies could you put wheels with this fix to https://test.pypi.org before making the official release? Anyway good work, as always! |
* Patch os.environ Resolves #673 * Update version * --gdal-data and --proj-data options for fio-env * Skip wheel tagged tests
As I was struggling with this stuff today ->
Permalink to the code as linenumbers change over time: Lines 321 to 323 in de6a122
whoa, github inlines that :O |
Hi, I just upgraded fiona to 1.8.4. I tried to load a shape file created last week (with release 1.7.13) and got these warnings: The loaded object doesn't have a crs, which it definitely had when I stored it. If I create a new file with 1.8.4 and then load it, the problem disappears. FYI, I'm using fiona through geopandas. I'm not using that many files, so I could recreate all of them with 1.8.4. But others might have a larger number of files, so I thought it'd be nice to report this issue. |
@xhoffmann you won't need to recreate files. It's only the crs property accessor that isn't working for you. Here's the deal: Fiona 1.7.13 patches os.environ to set GDAL_DATA and PROJ_LIB when we called with fiona.Env():
with fiona.open('example.shp') as collection:
print(collection.crs) or you can set a GDAL_DATA environment variable to point to the directory of support files. If you're using a Fiona wheel from PyPI, you can run |
We are observing some weird caching: $ python
>>> import fiona
>>> with fiona.open("test_id.shp", 'r') as source:
... print(source.crs)
...
ERROR 4: Unable to open EPSG support file gcs.csv. Try setting the GDAL_DATA environment variable to point to the directory containing EPSG csv files.
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
{}
>>> with fiona.Env():
... with fiona.open("test_id.shp", 'r') as source:
... print(source.crs)
...
{}
>>>
$ python
>>> import fiona
>>> with fiona.Env():
... with fiona.open("test_id.shp", 'r') as source:
... print(source.crs)
...
{'init': 'epsg:4326'}
>>> with fiona.open("test_id.shp", 'r') as source:
... print(source.crs)
...
{'init': 'epsg:4326'} This was a bit difficult to discover because it looked like non deterministic at first, but I just reproduced with a clean installation of the latest fiona wheel. |
@Juanlu001 can you share that shapefile with me? I can't reproduce the issue with files I have. |
@sgillies file that you asked about: test_id.tar.gz |
@sgillies was you able to reproduce the described behaviour using this file? |
@drnextgis @Juanlu001 I'm unable to reproduce the problem with fiona 1.8.4 on my macbook:
If GDAL_DATA is set in your environment and is not correct, you might see the symptoms you reported. Can you check that? @Juanlu001 there is indeed some SRS caching in GDAL. You can see signs of it in https://github.com/OSGeo/gdal/blob/master/gdal/ogr/ogr_fromepsg.cpp. I don't fully understand how it works, but I'm glad it's there so we don't have to read the support files over and over again. I assume it's going to change quite a lot when the new PROJ features land in GDAL 2.5. |
Sorry @drnextgis, I meant 1.8.4 and have edited my comment above. |
@sgillies I prepared some scripts and a dockerfile to reproduce the issue: https://github.com/Juanlu001/gdal-data-issues In any case, if the final word about this and rasterio/rasterio#1539 is "always use |
@Juanlu001 I think I may have explained the situation poorly. Setting GDAL_DATA and PROJ_LIB to point to the directory inside the installed wheel will always work. You can find these directories by using But if you don't want to set GDAL_DATA and PROJ_LIB in your production environment, you can execute code within an Env block and the directories will be found and GDAL will be configured within the block. |
I think I understand it in full now. Thanks for the explanation! |
Bumped into this issue after getting an
More details:
Whatever fixed it in In comments above, there is a
It would really help if
This venv also installs pyproj-3.0.1 that has another copy of binary libproj, so I wanted to check that version too:
There is a
package integrations and dependenciesThis issue is only about getting consistent information about the bundled libs and data (GDAL and PROJ), but it also begs questions about package integrations. Ignore the following if it represents too much scope creep. It seems like the short answer to the notes below is to build fiona and rasterio without binary wheels against OS packages for shared libs. This gets into how packaging tools expose options to manage all that (linux distros, conda, poetry, pip, etc etc). It seems like some kind of dummy package with integration tests could help to catch inconsistencies or some kind of abstraction for a shared package is required to manage the common dependencies and CLI tools for All the bundled shared libs:
|
This issue has the same symptoms as rasterio/rasterio#1539:
The open function is decorated by
ensure_env_with_credentials
but it seems doesn't work.The text was updated successfully, but these errors were encountered: