Validate external mapping size

[matvey-mtn]

• add checks for external mapping size
• ignore empty lines

[DanMelman]

Nice!
Small comments inside

[DanMelman]

can this be null while the file is still valid and big?

[matvey-mtn]

yes, the "Content-Length" header is not required by HTTP protocol.
I cannot assume it's always present.

[DanMelman]

So if the url is not configured to return this header, this validation is skipped.
So if you have a file with "only" 50,000 lines, but each line is more than 1MB, we will pass EXTERNAL_MAPPING_MAX_BYTES

[matvey-mtn]

yep, but there is nothing we can do to avoid this.

for s3 objects Content-Length is always present
for other cases we can instruct users to add Content-Lenght header to protect their systems

[DanMelman]

there are two things we can do:

1. do not allow urls without that header (makes things more complex for users, but protects our system better)
2. add some validation while iterating the file - like we count the lines, we can roughly count the bytes/chars per line

[matvey-mtn]

added fallback for counting length of the input if Content-Lenght is absent

[DanMelman]

I wouldn't do it as a fallback, I would just remove the header-based validation.
Remember that anyone can manually tweak that header and send you fake length value.
Also, this will keep things simpler.

[matvey-mtn]

uh, this will require adding ResponseTransformer to the tests. Hold on.

[DanMelman]

you anyway needed to test the fallback, no?

[DanMelman]

try finding a way to override the constants in the test

[DanMelman]

Looking very good!