Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use read_csv_to_dataframe in CSV validation #1379

Closed
emlys opened this issue Aug 22, 2023 · 0 comments · Fixed by #1419
Closed

Use read_csv_to_dataframe in CSV validation #1379

emlys opened this issue Aug 22, 2023 · 0 comments · Fixed by #1419
Assignees
Labels
enhancement New feature or request in progress This issue is actively being worked on

Comments

@emlys
Copy link
Member

emlys commented Aug 22, 2023

Validation has its own logic for reading and checking CSVs. This is now redundant to utils.read_csv_to_dataframe and less comprehensive. It would make sense to read CSVs for validation in the same way that we're going to read them in execute.

Currently, CSV validation check that:

  • the csv exists
  • the file has the necessary permissions
  • exact column names exist in the csv

With utils.read_csv_to_dataframe, we can also easily check that:

  • at least one column name matching a pattern exists in the csv
  • values can be cast to the appropriate data type for each column

related issues

This makes sense to do before #327 because read_csv_to_dataframe handles path expansion.

I'd also like to revisit whether we can drop support for Excel files (which is only allowed in one table in HRA) rather than writing additional code to support them.

@emlys emlys added the enhancement New feature or request label Aug 22, 2023
@emlys emlys changed the title Expand CSV validation using read_csv_to_dataframe Use read_csv_to_dataframe in CSV validation Aug 24, 2023
@emlys emlys self-assigned this Sep 25, 2023
@emlys emlys added the in progress This issue is actively being worked on label Sep 25, 2023
emlys added a commit to natcap/invest.users-guide that referenced this issue Nov 7, 2023
emlys added a commit to natcap/invest.users-guide that referenced this issue Nov 8, 2023
emlys added a commit to emlys/invest that referenced this issue Nov 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request in progress This issue is actively being worked on
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant