New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature]support latest version of Spark 3.2.0 #875
Comments
Hi @RickyHuo, |
Hi @RickyHuo, Thanks |
@ashmeet-kandhari Thanks for your contribution. I want to know the version compatibility with Spark lower version or Scala because I know Spark 3.X built with Scala 2.12. If we can use spark 2.4 local with seatunnel built on Spark 3.2 or not? |
@RickyHuo From what I see, there are API changes due to which this is not possible without adding some logic to look at the api |
How is it going ,in my opinion it's better to be compatible with both spark2 and spark3,spark version can be passed in maven as a parameter,and I can take some help if needed. |
Some connectors or class may need have different implement for spark2 and spark3. |
This method can‘t come true, because spark api have change between 2.x and 3.x. We can‘t maintain two version api in one project |
How is it going? @ashmeet-kandhari If you don't have enough time, I will do this recently. |
Hi @benjfan, I missed the updates recently. Thanks for offering :) |
@CalvinKirs Please assgin me |
support spark3.2.0, spark3.1.2 and spark2.4.7, please see https://github.com/nishuihanqiu/incubator-seatunnel with branch 2.1.3-spark3, that is 2.1.3-spark3 |
No description provided.
The text was updated successfully, but these errors were encountered: