Support for dataproc sessionTemplates in Vertex Workbench

[lgbaeza]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Description

A session template allows to run serverless interactive development through Dataproc Sessions in Jupyter notebooks, natively integrated with Vertex AI Workbench.

This feature is missing in Terraform, so WorkBench notebooks intended to run on top of Dataproc serverless, currently cannot be fully deployed with Terraform since users must create on their own the session templates (called runtime templates on the UI). The Dataproc API related to this resource is SessionTemplate

New or Affected Resource(s)

• google_workbench_instance

Potential Terraform Configuration

resource "google_dataproc_session_template" "runtime_template" {
  name = "dataproc runtime template"
  description = "Serverless dataproc session to be used on Vertex AI"
  runtimeConfig = {
    containerImage = "gcr.io/my-project/custom-spark:v0.0"
  }
  environmentConfig = {
    executionConfig = {
      serviceAccount = "dataproc-sa@my-project.iam.gserviceaccount.com"
      networkUri = "default"
      subnetworkUri = "default"
    }
  }
}

References

• Add resources for Vertex AI notebook runtime templates could look similar, however that issue is for the Workbench notebook runtime, this is for the dataproc runtime used by the notebook.

b/331218090
b/331217910

[melinath]

Note from triage: This looks likely to require changes to both dataproc and workbench, but it's not entirely clear how a user should proceed in this case.