feat: Add Apache Spark operator implementation #77
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add spark.py operator module with create/delete/patch functions - Add Kubernetes manifests (StatefulSet, Deployment, Services, RBAC) - Integrate Spark operator in main.py (create/delete handlers) - Extend Whisk CRD with Spark configuration schema - Add test configuration example (whisk-spark.yaml) Features: - Spark Master with optional HA support - Scalable Spark Workers (1 to N replicas) - History Server with persistent storage - Complete RBAC configuration - Health checks and resource management - Event logging support Implements all OpenServerless operator patterns: - Kopf framework handlers - Owner references for garbage collection - Kustomize templating - Status tracking in CRD - Error handling and logging
- Implement kopf-based SparkJob operator handlers (create/delete) - Add SparkJob CRD definition for nuvolaris.org/v1 - Create Kubernetes Job template for Spark driver execution - Configure GHCR authentication for mobs75/openserverless-operator:spark-dev - Add operator pod configuration with proper resource allocation - Update build tasks for Spark operator deployment
- Add Spark cluster deployment templates (master, worker, history) - Update Spark StatefulSets and Deployment configurations - Add Spark ConfigMap and RBAC templates - Update kaniko build and operator deployment configs - Update README with Spark operator documentation
- Remove TaskfileBuild.yml, TaskfileDev.yml, TaskfileOlaris.yml, TaskfileTest.yml - These belong to the task project, not operator project - Keep main Taskfile.yml for operator-specific tasks
- Add Spark cluster deployment with master + 2 workers + history server - Implement SparkJob CRD support (sparkjobs.nuvolaris.org) - Configure NodePort services for Web UI external access - Optimize memory settings for single-node MicroK8s deployment - Add Spark component tracking in main.py and patcher.py - Include test configuration whisk-spark-test.yaml - Update operator image to ghcr.io/mobs75/openserverless-operator:dev-spark-v6 Tested features: ✅ Complete Spark cluster deployment and operation ✅ SparkJob CRD creation and management ✅ Web UIs accessible via NodePort (Master 30808, Workers 31081/31082, History 31808) ✅ Job submission and driver execution
- Modifica worker Spark - Fix handler spark.py - Aggiorna operator-deploy.yaml per Spark Branch: feature/add-spark-operator
mobs75
force-pushed
the
feature/add-spark-operator
branch
from
afc74b4 to
a2f3dae
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Complete Apache Spark operator implementation for OpenServerless.
Changes
New Files
nuvolaris/spark.py: Core Spark operator module (create/delete/patch)nuvolaris/templates/spark-*: Kubernetes manifests (Master, History Server, RBAC)deploy/spark/: Deployment resourcesModified Files
nuvolaris/patcher.py: Added Spark patching logicnuvolaris/main.py: Integrated Spark into operator workflowKey Features
1Gi↔ JVM1g)Testing
Verified on MicroK8s:
Part of main integration PR: apache/openserverless#TBD