Limit feature file size

[AhmedGrati]

What would you like to be added:
Currently, we are processing feature files without any restriction on the file size, since we are expecting that the file is coming from a trusted source. However, processing large files can present some bottlenecks. Therefore, we should limit the size of features' files.
For more information: #1285 (comment)

Why is this needed:
Improve confidence in the system, by covering edge cases concerning feature file size.

[AhmedGrati]

FYI @marquiz.
BTW I have two questions concerning this:
1- Should we make the file size limit configurable, with a default value of 64kB, so users can adjust the limit based on their use cases?
2- If the size of a file is bigger than the limit, should we process it in chunks, or raise an error to the user?

[marquiz]

Thanks @AhmedGrati for creating this issue

1- Should we make the file size limit configurable, with a default value of 64kB, so users can adjust the limit based on their use cases?

I don't think we need to make it configurable. I believe 64kB (or 32kB) should be plenty, enough. Let's revisit this if somebody really comes up with a use case for bigger feature files.

2- If the size of a file is bigger than the limit, should we process it in chunks, or raise an error to the user?

I would consider it as a "transaction", all or nothing, not advertise partial results. I.e. skip the file if it's too big

ping @eero-t

[eero-t]

It's possible that some feature files could in addition to labels, include extensive documentation for those labels as comments, which I think to be a valid use-case.

Main things that should IMHO be caught is accidentally giving NFD the old binary hook as feature file, or things intentionally trying to break NFD.

=> I think the limit can be fixed size, fairly big, and documented [1]. 64kB sounds fine, but it could be even larger.

I would consider it as a "transaction", all or nothing, not advertise partial results. I.e. skip the file if it's too big

Agree, log an error [2] instead of parsing file, and move to next file.

[1] E.g: "NFD logs an error and skips feature files larger than XX KiB. Larger files are assumed to be configuration errors."
[2] "Skipping feature file '%s' with invalid size (%d KiB > max %d KiB)"

[AhmedGrati]

/assign