Conversation
This filter is needed for the FeedParser topology, but making it more generic has no cost, so we end up with a filter that looks for some user-defined strings in a record metadata URL key, such as records read with the Files input. If one of the user-defined strings is found in the URL, this new filter will sets an user-defined field to that string. Records for which the URL doesn't contain any of the strings are discarded.
| Strings []string `help:"Strings to look for in the URL. Discard records not containing any of them." required:"true"` | ||
| } | ||
|
|
||
| func (cfg *setStringFromURLConfig) fillDefaults() error { |
There was a problem hiding this comment.
maybe in another PR we could make a general refactoring renaming these fillDefaults() to checkConfsAndFillDefaults() as this is what we do (not only here, everywhere)
There was a problem hiding this comment.
Yeah, good point. I'd like us to discuss a way to enforce that pattern in all components, maye by making the config interfaces actual type Config interface{ Validate() error } or something, rather than mere interface{}.
Id also like us to think about going further and find an easy way to do that as well as providing an easy way to document required field and default fields but also make applying the defaults, validating the values and providing help all at once.
tommyblue
left a comment
tommyblue
left a comment
There was a problem hiding this comment.
Nor approving neither rejecting because I don't know if the multiple calls to next is what you want. If it is, then it's approved.
If it's not (i.e. the next must be moved after the for loop, then the l.Set() call is wrong because it will overwrite the previous loop value
2 participants
❓ What
This filter is needed for the FeedParser topology, but making it more generic has no cost, so we end up with a filter that looks for some user-defined strings in a record metadata URL key, such as records read with the Files input. If one of the user-defined
strings is found in the URL, this new filter will sets an user-defined field to that string. Records for which the URL doesn't contain any of the strings are discarded.
🔨 How to test
✅ Checklists
This section contains a list of checklists for common uses, please delete the checklists that are useless for your current use case (or add another checklist if your use case isn't covered yet).
make gofmt-writebeen run on the code?make govetbeen run on the code? Has the code been fixed accordingly to the output?