feat: Add AWS SQS Sink functionality #1
Conversation
Signed-off-by: chandankumar4 <chandan.kr404@gmail.com>
Signed-off-by: chandankumar4 <chandan.kr404@gmail.com>
Signed-off-by: chandankumar4 <chandan.kr404@gmail.com>
chandankumar4
force-pushed
the
aws-sqs-sink
branch
from
10b3441
to
ca8dcb6
Compare
.github/workflows/ci.yaml
Outdated
| - name: "Step 6: Load image in cluster" | ||
| run: | | ||
| docker load --input /tmp/aws-sqs-queue.tar | ||
| kind load docker-image "quay.io/numaio/numaproj-contrib/aws-sqs-sink-go:sqs-e2e" |
main.go
Outdated
| if !ok { | ||
| logger.Fatalln("AWS_REGION not found") | ||
| } | ||
| accessKey, ok := os.LookupEnv("AWS_ACCESS_KEY") |
There was a problem hiding this comment.
The credentials should not be mandatory, ppl might use things like IAM Role for service account to access.
main.go
Outdated
| if _, err = s.sqsClient.SendMessage(ctx, &sqs.SendMessageInput{ | ||
| MessageBody: &msgBody, | ||
| QueueUrl: queueURL.QueueUrl, | ||
| }); err != nil { | ||
| s.logger.Errorf("failed to push message %v", err) | ||
| continue |
There was a problem hiding this comment.
invoking send for each message might be expensive.. Consider writing in batches?
Signed-off-by: chandankumar4 <chandan.kr404@gmail.com>
Signed-off-by: chandankumar4 <chandan.kr404@gmail.com>
Signed-off-by: chandankumar4 <chandan.kr404@gmail.com>
| limitations under the License. | ||
| */ | ||
|
|
||
| package fixtures |
There was a problem hiding this comment.
we should extract one out for contrib org.
Signed-off-by: chandankumar4 <chandan.kr404@gmail.com>
Signed-off-by: chandankumar4 <chandan.kr404@gmail.com>
Signed-off-by: chandankumar4 <chandan.kr404@gmail.com>
|
Sry one more comment, are we suppose to see CI workflow succeeds? I don't see checks running for this PR. |
yaa, I think it will trigger after first PR merge to main branch with workflow yaml but not sure (that's what I have observed in previous repo) maybe someone need to enable it manually? |
|
Also unit test case is pending, I'm working on that. probably we merge it also and I'll raise another PR for it? |
Sounds good. I am approving this PR. Feel free to merge it and use another PR to trigger CI. |
|
I don't have access to merge ;) |
Done. |
Add the functionality to sink pipeline vertex data to aws queue. Please check README for example pipeline
Fixes: numaproj/numaflow#1093