You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the feature you'd like
It would be nice to have the possibility to choose different instance types for the driver node and the worker nodes when using the PySparkProcessor.
Describe alternatives you've considered
It is possible to choose a high-memory instance for all instances but it could be unnecessarily costly for the user.
Additional context
Some pyspark operations (e.g. .toPandas()) are memory expensive for the driver node.