Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SEDONA-500] Fix failures and data consistency problems when reading multiple shapefiles in one directory #1250

Merged
merged 1 commit into from
Feb 22, 2024

Conversation

Kontinuation
Copy link
Member

Did you read the Contributor Guide?

Is this PR related to a JIRA ticket?

What changes were proposed in this PR?

Fix problems when generating splits for directories containing multiple shapefiles. This fixes the following issues:

  1. Reading fails with errors such as "shape record loses attributes in .dbf file at ID=2387"
  2. Geometry values and other fields do not match.
  3. Not all records were read into SpatialRDD

How was this patch tested?

Add additional assertion to the testReadMultipleShapeFilesByMultiPartitions test case.

Did this PR include necessary documentation updates?

  • No, this PR does not affect any public API so no need to change the docs.

@Kontinuation Kontinuation marked this pull request as ready for review February 22, 2024 02:52
@jiayuasu jiayuasu merged commit 291fd96 into apache:master Feb 22, 2024
46 checks passed
jiayuasu pushed a commit that referenced this pull request Apr 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants