Skip to content

Conversation

rohkhann
Copy link
Contributor

@rohkhann rohkhann commented May 21, 2024

Why make this change?

  1. Without adding limits on the amount of data that can be read from the db, the current reads could overload the memory of the hosting infrastructure of data-api-builder. Because of this, we need to have a limit on the amount of data that can be pulled from memory.

What is this change?

  1. This change uses Sequential access and buffers to buffer data in from the db.
  2. It throws exception if - there is more data available to read from the db for this particular query but we have already read the max buffer size available.

How was this tested?

  • Integration Tests
    All existing integration tests execute successfully ensuring new code is only executed if feature flag is enabled.
  • Unit Tests
    added unit test for streaming logic.

@rohkhann
Copy link
Contributor Author

rohkhann commented Jun 6, 2024

/azp run

@rohkhann rohkhann changed the title Add a max response size property and stream results from db to ensure application memory is protected in dab. Streaming results in query flow to protect application memory Jun 6, 2024
@rohkhann
Copy link
Contributor Author

rohkhann commented Jun 6, 2024

/azp run

@rohkhann rohkhann requested a review from seantleonard June 6, 2024 18:12
@seantleonard
Copy link
Contributor

Unclear which Integration tests apply to this PR as mentioned in PR description:

Integration Tests

rohkhann and others added 4 commits June 10, 2024 12:24
Co-authored-by: Sean Leonard <sean.leonard@microsoft.com>
Co-authored-by: Sean Leonard <sean.leonard@microsoft.com>
@rohkhann rohkhann requested a review from seantleonard June 11, 2024 17:06
@rohkhann
Copy link
Contributor Author

/azp run

@rohkhann rohkhann enabled auto-merge (squash) June 11, 2024 23:19
@rohkhann
Copy link
Contributor Author

/azp run

@rohkhann rohkhann merged commit 8f8f53f into main Jun 12, 2024
@rohkhann rohkhann deleted the rohkhann/EnforceDatastreamingLimits branch June 12, 2024 00:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants