Fix closing connection before content is read

[ace-e4s]

This PR is related to user story ESS-XXXX

Description

Fixes an issue in _blob_to_df, where connection was set to stream=True and closed, before content was read. This meant that the connection was available in the pool while the data was still being read.

If the connection is then reused by someone else, this would look like the connection broke from server side, potentially leading to all the ChunkedEncodingError and IncompleteReadError.

Fixed by reading the content all at once while connection was still open. No need for streaming, since we dont use the partial data.

Checklist

• PR title is descriptive and fit for injection into release notes (see tips below)
• Correct label(s) are used

PR title tips:

• Use imperative mood
• Describe the motivation for change, issue that has been solved or what has been improved - not how
• Examples:
  • Add functionality for Allan variance to sensor_4s.simulate
  • Upgrade to support Python 9.10
  • Remove MacOS from CI

[bjorn-einar-bjartnes-4ss]

Well spotted @ace-e4s . I think even better would be to put everything inside the with statement and pass the stream directly on without any intermediate copying, that should cause less memory allocation. These files can be quite large. I will try to come up with a suggestion.

[ace-e4s]

Well spotted @ace-e4s . I think even better would be to put everything inside the with statement and pass the stream directly on without any intermediate copying, that should cause less memory allocation. These files can be quite large. I will try to come up with a suggestion.

That is a good point. I principle, we could just send response.raw directly into read_csv. However, there is one show stopper.

DRIO may send "malformed" CSV, which requires us to define the seperator as a regex and use a pure Python csv engine. This has a huge(!) performance impact. To mitigate this, we have choosen to inspect the entire content (counting new lines and commas) and determine if the csv can be parsed using regular C engine, and only use the Python engine as a last resort.

The price we pay is double memory usage for a few seconds while parsing.

Malformed csv: DRIO assumes everything is 2 column csv. Therefore, it may send a csv file where there is more than 1 comma, and no quatation is used to encapsulate the second column.

Example of a malformed line: `1234, {"key1": "value1", "key2": "value2"}

[ace-e4s]

So, if DRIO operated with RFC4180 definition of CSV (https://datatracker.ietf.org/doc/html/rfc4180.html), we could have done it more memory efficient and always used the fastest parser.

[bjorn-einar-bjartnes-4ss]

Thanks, this is nice to know. So when the value is JSON we can have commas in there, therefore parsing it requires special care. JSON and CSV sounds like a strange mix, we would have to send some encoded JSON or something to make that work, I would assume. This is indeed a strange mix...

I guess we will have to continue doing it for now, but we should discuss this with the DRIO team. It shows how some of these assumptions about values being a string has consequences.

[bjorn-einar-bjartnes-4ss]

We should have something in the code that explains the intent behind the code that handles the parsing, a comment is probably good enough.