Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pulsar - Spark adapter for scala 2.11 #29

Open
aditiwari01 opened this issue Dec 10, 2021 · 8 comments · May be fixed by #31
Open

Pulsar - Spark adapter for scala 2.11 #29

aditiwari01 opened this issue Dec 10, 2021 · 8 comments · May be fixed by #31

Comments

@aditiwari01
Copy link

The pulsar-spark adapter available seems to be using spark-streaming-2_10.
Is there any plan for adapter for spark_2_11?

@eolivelli
Copy link
Contributor

There is no much activity on the Spark adapter.

Would you like to send a patch ?

@aditiwari01
Copy link
Author

I'm exploring options as of now. If I end up writing a patch, would definitely send it.

@aditiwari01
Copy link
Author

The current available spark pulsar adapter seems to be unreliable. I've written a reliable connector with addition consideration for back-pressure and rate limit. Currently testing around it. Can raise a PR if that's fine.

@eolivelli
Copy link
Contributor

Can raise a PR if that's fine

Sure! thanks

@aditiwari01
Copy link
Author

aditiwari01 commented Dec 26, 2021

@eolivelli
I have raised a PR with a new receiver. I'm yet to add examples/documentation for the same, but would appreciate if you can go through it and provide the review.

@aditiwari01
Copy link
Author

Also can you approve running workflows so I can check if there are any issues.

@aditiwari01
Copy link
Author

@eolivelli Need a small help. In local I've been building with skip License. Flow is failing because of
Some files do not have the expected license header

How can I add license to the new files that I have added?

@michaeljmarshall
Copy link
Member

@aditiwari01 - try running mvn initialize license:format from the base directory of the project. Let me know if that doesn't work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants