Improve data loading error message#55
Conversation
There was a problem hiding this comment.
That's very useful, Mike - thank you for working on this!
Would it be ok to add a hint to the operator to why the dataset is empty when they know it's certainly isn't empty?
e.g. perhaps adding something like "Empty dataset after filtering out data that doesn't match the set requirements, e.g. max_length is smaller than the length of the shortest record"
Perhaps it's too verbose and can be phrased more succinctly but it'd help to explain like you did on slack for me.
Sure I think we could add these into the actual filtering step for SFTDataFactory. Let me make those changes now! |
sfc-gh-sbekman
left a comment
There was a problem hiding this comment.
This is great now, thank you, Michael!
With some data loading configurations, it's possible to filter out all of the dataset during data loading. This leads to ambiguous error messages later. Adding checks here that raise more descriptive errors when we encounter this problem.