-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
connect_hub() behaves differently for invalid model-output files in parquet format (vs .csv files) #33
Comments
Good spot! Unfortunately this seems to be something emanating from With With While it would definitely be nice to have consistent behaviour for a user, I'm not sure how easy/straightforward it will be to implement given the lower level difference in behaviour and will need some exploring. At least an error is being thrown at some point (while for csvs the data was just excluded without notice), but it certainly is not as helpful as the csv one. That's also why the validation stage is such an important part of the hubs. Because arrow can be a bit brittle and the error messaging is not super helpful yet, it's much more efficient to ensure data integrity on the way in than try and do too much at runtime when accessing the data. |
I had that error with |
Ah, I didn't realize that the reason hubData has that nice check for the .csvs is because Arrow was silently excluding them! @annakrystalli I think it's completely reasonable not to invest a lot of time doing downstream checks and workarounds to accommodate hub data that doesn't pass the upstream validations. Especially since, as you say, arrow throws an error in this case (i.e., users won't unknowingly get incorrect information). If we were going to invest time, I'd argue that it would be better spent upstream, helping people submit correct data to begin with. I'm happy to close this one as a "wontfix" |
I came across this discrepancy when testing S3-based FluSight parquet model-output files and was able to reproduce the behavior on my local machine.
The discrepancy involves how hubData handles connections to model-output data when there's a model-output file with invalid data.
Not sure if:
Also, the tasks.json for this hub specify reference_date and target_end_date in YYYY-MM-DD format, so I'm not sure how much work we should do to accommodate this example of an invalid file.
Happy to move the issue elsewhere if necessary!
Background
The FluSight hub contains one model-output file with invalid date formats (
reference_date
andtarget_end_date
are3/2/2024
instead of2024-03-02
)..csv example
hubData handles the situation very gracefully when using it against the .csv model-output files on my local machine. Essentially, it warns about an invalid file, keeps going, and still produces a tibble.
parquet example
When using hubData to connect to model-output files we converted to parquet format in the cloud, there's no warning, but trying to see the tibble produces an error.
try local connection using the same converted parquet files
To rule out a difference between connect locally vs connecting to S3, I copied the converted parquet files to my local machine and tried again to connect. Result was the same error as connecting to S3.
The text was updated successfully, but these errors were encountered: