Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Apache NiFi integration support #98

Merged
merged 14 commits into from
Nov 28, 2023
Merged

Conversation

kmushegi
Copy link
Contributor

@kmushegi kmushegi commented Nov 28, 2023

What it Does

  • Add the aws-ignore-auth-header-region flag to serve command
  • Add the matching field to Bolt router config
  • If the above field is true set bucket region to the region from bolt vars instead of inferring from the Credential part of the Authorization header.
  • Add a bad request helper function and return bad request if request incoming to Sidekick does not have an authorization header, instead of returning a 500.
  • Add apache nifi integration docs

Why

This change is motivated by the integration with Apache NiFi. When NiFi *S3 processors make a request to Sidekick the Authorization header of the incoming request looks like

Authorization: AWS4-HMAC-SHA256 Credential=AKIA5M5ADDSG4RWJDKUS/20231128/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;user-agent;x-amz-content-sha256;x-amz-date, Signature=9734a673ce1febada676904f1006c315737b3e451f570ca4dfea0167ab6c0ab3

The interesting part here is Credential which we use to deduce the region to use when creating requests going to Crunch and AWS. Despite configuring the regions in each processor (and some under the hood NiFi configs), the Credential part looks like

AKIA5M5ADDSG4RWJDKUS/20231128/us-east-1/s3/aws4_request

and the inferred region is us-east-1 while the bucket actually resides in us-west-2.

After this change, we're able to force the region specified via the GRANICA_REGION env var explicitly and get NiFi working

@kmushegi kmushegi marked this pull request as ready for review November 28, 2023 01:06
@kmushegi kmushegi changed the title feat: option to not infer region from the Authorization header in AWS mode feat: Apache NiFi integration support Nov 28, 2023
@kmushegi kmushegi merged commit 9649eb2 into main Nov 28, 2023
7 checks passed
@kmushegi kmushegi deleted the feat/ignore-auth-header-region branch November 28, 2023 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants