Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use same Json schema as Java library #19

Closed
dbatyuk opened this issue Dec 2, 2020 · 6 comments
Closed

Use same Json schema as Java library #19

dbatyuk opened this issue Dec 2, 2020 · 6 comments
Labels
enhancement New feature or request

Comments

@dbatyuk
Copy link

dbatyuk commented Dec 2, 2020

Is it possible to use same schema as
https://docs.aws.amazon.com/sns/latest/dg/large-message-payloads.html
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html
java libraries for SNS and SQS to be able to produce/consume using nodeJs and consume/produce using Java.

they use next Json
[ "software.amazon.payloadoffloading.PayloadS3Pointer", { "s3BucketName": "extended-client-bucket", "s3Key": "xxxx-xxxxx-xxxxx-xxxxxx" } ]

or at least maybe make it configurable

Thanks

@mzahor
Copy link
Contributor

mzahor commented Dec 17, 2020

It's a good suggestion, but I don't think we can change our current format, because existing consumers would break (unless we do some versioning which is even more complicated). I think the way to go is to make it configurable. Unfortunately it's not our focus atm, but feel free to sumbit a PR.

@mzahor mzahor added the enhancement New feature or request label Dec 17, 2020
@FrederikP
Copy link
Contributor

Stumbled over the same issue.
We use some components written in Java that use the SQS Extended library (by awslabs) or python that use compatible libraries, but we didn't find anything compatible for node.

I implemented an opt-in compatibility mode here: https://github.com/FrederikP/sns-sqs-big-payload/tree/introduce-awslabs-compatibility-optin

It's hard to write an integration test because that would mean creation java resources etc. in this project.
For now I'll just test in our dev environment. I'll let you know how it goes.

Feel free to suggest changes and I will gladly open up a PR if it helps.
Thanks for this library ;)

@FrederikP
Copy link
Contributor

Okay we tested the compatibility mode in our environment. (sending in non nodejs component, receiving in nodejs component) and it works.
Is someone interested in using this? I can open up a PR.

@mzahor
Copy link
Contributor

mzahor commented Feb 4, 2021

Hi @FrederikP, sounds great, please open a PR.

@FrederikP
Copy link
Contributor

Ok. See #26
Feel free to change stuff as you see fit.

@mzahor
Copy link
Contributor

mzahor commented Feb 8, 2021

Released in v0.1.0

@mzahor mzahor closed this as completed Feb 8, 2021
YanivD pushed a commit that referenced this issue Feb 13, 2022
* fix: use correct schema for extended lib compatibility

Fixes S3 reference in the message body not matching with the JSON schema of the Amazon SQS Java Extended Client library.

fix #42 #19

* test: add test for producing messages in compatibility mode

* refactor: align s3extendedPayloadError with the error schema

* docs: document s3-extended-payload-error event
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants