Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

coshp utility to support "cloud optimized shape file" #6518

Merged
merged 3 commits into from
Jun 24, 2022

Conversation

pramsey
Copy link
Contributor

@pramsey pramsey commented Apr 20, 2022

To enable fast bounding box access of shapes over HTTP-offset reads, we need to sort the contents of shape files so that they appear in the same order in the shp file as they do in the qix spatial index. (Also the contents of the dbf need similar sorting.

In the spirit of sortshp the coshp utility takes in an unsorted shape file, and writes out a sorted shape file (shp, shx, dbf, qix).

> coshp                                   
Syntax: coshp [infile] [outfile]

> coshp va_2013_carto va_2013_carto_sorted
Wrote 9349 spatially sorted shapes into 'shapefile va_2013_carto_sorted'

@jmckenna
Copy link
Member

Thanks @pramsey . Is it possible for you to also include an msautotest/test as part of this PR for this utility? (see https://mapserver.org/development/tests/autotest.html )

@jmckenna jmckenna added this to the 8.0 Release milestone Apr 20, 2022
@pramsey
Copy link
Contributor Author

pramsey commented Apr 21, 2022

This doesn't seem to be standard practice for the utility programs? I cannot find any tests running for shptree or sortshp.

@jmckenna
Copy link
Member

I'm not sure about 'standard practice', but I think for a new utility as part of the MapServer 8.0 release the new utility should be tested as part of msautotest.

Hope that answers your question.

@rouault
Copy link
Contributor

rouault commented Apr 21, 2022

I cannot find any tests running for shptree or sortshp.

true, we are lacking on that side.
Potential way forward:

  • add a msautotest/utils directory with a 0-byte __init__.py file
  • referencing that new directory in testpaths of msautotest/pytest.ini
  • add a msautotest/utils/test_coshp.py file importing pytest, having one or several test functions (prefixed by "test_") and using os.system() to run coshp and doing some after checks
  • add a utils-testcase target in top Makefile that just cd msautotest/utils and run pytest in it

@sdlime
Copy link
Member

sdlime commented Apr 22, 2022

I cannot find any tests running for shptree or sortshp.

true, we are lacking on that side. Potential way forward:

  • add a msautotest/utils directory with a 0-byte __init__.py file
  • referencing that new directory in testpaths of msautotest/pytest.ini
  • add a msautotest/utils/test_coshp.py file importing pytest, having one or several test functions (prefixed by "test_") and using os.system() to run coshp and doing some after checks
  • add a utils-testcase target in top Makefile that just cd msautotest/utils and run pytest in it

Seems like reasonable tests would be:

For sortshp and coshp:

  1. compare a source and output shapefile properties e.g. feature count, extent.
  2. compare maps rendered from source and output shapefiles.

For shptree:

  1. compare maps rendered from a source shapefile with and without an index.

I'd can take a shot at @rouault suggestion for sortshp I guess.

@jmckenna
Copy link
Member

Should I tackle the msautotests for this? (beta release will be this coming Friday) Thanks!

@sdlime
Copy link
Member

sdlime commented Jun 24, 2022

IMHO, I don't think the tests should hold up the beta. I'd recommend a merge and then create a separate ticket related to the tests... --Steve

@jmckenna
Copy link
Member

thanks @pramsey ! merging now...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants