What is the problem the feature request solves?
We currently have one profile per Spark version e.g. spark-4.1. I'd like to explore having a spark-latest profile that pins to a recent commit in Spark's master branch. We could increment this periodically and catch issues earlier.
There are no JAR files Maven, so this would require some extra work to build Spark locally. It may be too expensive to run in CI.
I am not sure if this will be more trouble that it is worth, but I thought it would be wort exploring at least.
Describe the potential solution
No response
Additional context
No response
What is the problem the feature request solves?
We currently have one profile per Spark version e.g.
spark-4.1. I'd like to explore having aspark-latestprofile that pins to a recent commit in Spark's master branch. We could increment this periodically and catch issues earlier.There are no JAR files Maven, so this would require some extra work to build Spark locally. It may be too expensive to run in CI.
I am not sure if this will be more trouble that it is worth, but I thought it would be wort exploring at least.
Describe the potential solution
No response
Additional context
No response