Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update state and county shapefiles #3293

Merged
merged 4 commits into from
Nov 28, 2023

Conversation

dopplershift
Copy link
Member

@dopplershift dopplershift commented Nov 28, 2023

Description Of Changes

This started as a way to address warnings I started seeing with GDAL >= 3.7.3:

Warning 1: us_counties_20m.shp contains polygon(s) with rings with invalid winding order. Autocorrecting them, but that shapefile should be corrected using ogr2ogr for example.

It turns out these are actually spurious (see OSGeo/gdal#8767) and should be fixed with GDAL >=3.8.1.

In the course of investigating, I found that:

  1. Our files (esp. county files) are out of date. The files are from the Census Bureau 2016 collection and a few border and metadata updates have taken place.
  2. The metadata (i.e. attributes) on the state file is completely bogus; for instance, the name of a given state corresponds to one of (probably last/first) counties in the state.

Based on Shapely, it looks like the state borders we had were completely geometrically equivalent to the US Census Bureau state file collection from 2016. Therefore, this updates to use the U.S. state and county shapefiles from the 2022 collection. In the process, a few more files (which are downloaded automatically) are added that include the text encoding as well as a projection (though not yet used by Cartopy SciTools/cartopy#2289); as a result, this also updates the default globe for the feature to use the correct NAD83 rather than WGS84.

@dopplershift dopplershift added the Type: Maintenance Updates and clean ups (but not wrong) label Nov 28, 2023
@dopplershift dopplershift added this to the November 2023 milestone Nov 28, 2023
@dopplershift dopplershift requested a review from a team as a code owner November 28, 2023 00:56
@dopplershift dopplershift requested review from dcamron and removed request for a team November 28, 2023 00:56
Original county files were from the 2016 US Census Bureau collection.
The state files were generated separately using the county files, but
the geometry was completely topologically equivalent to the US Census
Bureau state file collection, so we now use the state collections
directly; the metadata (e.g. feature names) were incorrect in our
files and are now corrected.
Also set a correct default ellipse for these data, which are on NAD83
rather than WGS84.
We could actually fully automatically generate this, either as a
pre-commit hook or as part of building the wheel. Will hold off on those
until we are ready to do more.
@dopplershift
Copy link
Member Author

I added a step to our linting check to make sure the registry (static-data-manifest.txt) is up-to-date, since I neglected to do that early on here. It would be nice to auto-generate this since pooch has make_registry() to do this for us. Doing it at wheel-build time would make some sense, but we do still need the registry even for git checkouts; it probably makes the most sense to add a git pre-commit hook to regenerate it. This is probably most easily done when we finally wire up pre-commit.

@dcamron
Copy link
Member

dcamron commented Nov 28, 2023

Looks like we'll need some test updates. Seems appropriate to re-generate some of the test_cartopy_utils images to reflect these updated shapefiles. Most of the declarative failures could survive a tolerance bump, but in test_declarative_additional_layers_plot_options it seems like some of the boundaries are switching between 'dotted' and 'dashed' linestyles? Otherwise it is mostly just shifting contours.

@kgoebber
Copy link
Collaborator

The original US state files were generated from dissolving the counties by states using GIS, so that was why they were equivalent and also had not great metadata. I was not able to locate all of the files linked in the description of changes above, so I made my own. This PR is a vast improvement over the original files.

Copy link
Collaborator

@kgoebber kgoebber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good to me.

@dopplershift dopplershift merged commit 70e4faf into Unidata:main Nov 28, 2023
34 checks passed
@dopplershift dopplershift deleted the update-shapes branch November 28, 2023 23:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Type: Maintenance Updates and clean ups (but not wrong)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants