filter: Restore automatic boolean conversion #1410
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #1410 +/- ##
==========================================
+ Coverage 67.38% 67.46% +0.08%
==========================================
Files 69 69
Lines 7465 7484 +19
Branches 1836 1840 +4
==========================================
+ Hits 5030 5049 +19
Misses 2161 2161
Partials 274 274 ☔ View full report in Codecov by Sentry. |
victorlin
force-pushed
the
victorlin/fix-filter-query-bool
branch
from
a3fada4
to
df73174
Compare
Use one comment to explain both the example file and command. This will make the file more readable with multiple commands.
Previous wording was wordy. Make it more concise.
victorlin
force-pushed
the
victorlin/fix-filter-query-bool
branch
from
df73174
to
6765b8b
Compare
joverlee521
requested changes
Feb 9, 2024
There was a problem hiding this comment.
Thanks for the quick fix @victorlin!
I think there's a potential bug if user specifies --query-column with a numeric type. Otherwise, changes look good to me.
Since filter can now automatically infer boolean types, should bool be added to the ACCEPTED_TYPES for the --query-columns flag? Maybe outside of the scope of this PR, but just wondering if there are plans to add it later.
Automatic conversion is applied by default. Reserve the 'numeric' type for automatic conversions and keep the accepted values to --query-columns strict (for numeric, either 'int' or 'float').
Boolean conversion was not considered when automatic nullable numeric conversion was applied¹, but it continued to work because augur filter relied on pandas.read_csv's automatic type inference up until it was disabled in favor of reading all columns as string². A note to include boolean was added³ but removed inadvertently in another big change⁴. ¹ b325b97: Try converting all columns to numerical type ² 9f9be3a: Read all metadata as string type ³ 725e1b4: Expand comment on numeric conversion ⁴ b0a0d11: Add --query-columns option
This was not previously supported by pandas.read_csv's built-in type inference, but it aligns with the existing support for nullable numeric columns.
victorlin
force-pushed
the
victorlin/fix-filter-query-bool
branch
from
6765b8b
to
d44506d
Compare
joverlee521
approved these changes
Feb 10, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of proposed changes
Boolean conversion was not considered when automatic nullable numeric conversion was applied¹, but it continued to work because augur filter relied on pandas.read_csv's automatic type inference up until it was disabled in favor of reading all columns as string².
A note to include boolean was added³ but removed inadvertently in another big change⁴.
¹ b325b97: Try converting all columns to numerical type
² 9f9be3a: Read all metadata as string type
³ 725e1b4: Expand comment on numeric conversion
⁴ b0a0d11: Add --query-columns option
Related issue(s)
Checklist