Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support maximum request length configurations in the HTTP and OTel sources #3931

Closed
dlvenable opened this issue Jan 9, 2024 · 2 comments · Fixed by #3958
Closed

Support maximum request length configurations in the HTTP and OTel sources #3931

dlvenable opened this issue Jan 9, 2024 · 2 comments · Fixed by #3958
Assignees
Labels
enhancement New feature or request plugin - source A plugin to receive data from a service or location.
Milestone

Comments

@dlvenable
Copy link
Member

dlvenable commented Jan 9, 2024

Is your feature request related to a problem? Please describe.

Some configurations will not work if a request size is too large. For example, an http source with a kafka buffer will fail to write to Kafka if the request size is greater than 1MB. Data Prepper does not check the request size up-front leading to a confusing error.

Describe the solution you'd like

Provide new configurations on the HTTP/gRPC sources (this includes all three OTel sources) to configure a maximum request length.

e.g.

apache-log-pipeline:
  source:
    http:
      path: "/${pipelineName}/logs"
      max_request_length: 1mb

Describe alternatives you've considered (Optional)

Have different exceptions thrown from the buffer to indicate the size is too large.

Additional context

This is somewhat related to the PR to increase the Kafka message size in the kafka buffer:

#3916

@dlvenable dlvenable added untriaged enhancement New feature or request plugin - source A plugin to receive data from a service or location. and removed untriaged labels Jan 9, 2024
@dlvenable dlvenable added this to the v2.7 milestone Jan 9, 2024
@dlvenable
Copy link
Member Author

We should be able to do this:

sb.maxRequestLength(sourceConfig.getMaxRequestLength());

dlvenable added a commit to dlvenable/data-prepper that referenced this issue Jan 12, 2024
…ource, otel_metrics_source, and otel_logs_source sources. Resolves opensearch-project#3931

Signed-off-by: David Venable <dlv@amazon.com>
dlvenable added a commit that referenced this issue Jan 12, 2024
…ces (#3958)

Adds max_request_length as a configuration for the http, otel_trace_source, otel_metrics_source, and otel_logs_source sources. Resolves #3931

Signed-off-by: David Venable <dlv@amazon.com>
@dlvenable
Copy link
Member Author

These Data Prepper sources now return 413 Request Entity Too Large if the request length exceeds the maximum defined size. By default, there is no defined size.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request plugin - source A plugin to receive data from a service or location.
Projects
Development

Successfully merging a pull request may close this issue.

2 participants