Add support for datastream resources

[jtjanchor]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

I would love to see support for the following Datastream resources:

• streams
• private connections
• connection profiles

Datastream is generally available as of November 30, 2021. It would be great to have the ability to manage datastream resources via Terraform.

New or Affected Resource(s)

The new resources might look something like this:

• google_datastream_stream
• google_datastream_private_connection
• google_datastream_connection_profile

Potential Terraform Configuration

This is not exhaustive, but resources might look something like this:

resource "datastream_stream" "my_stream" {
  name = "my-stream"
  display_name = "My Stream"
  source_connection_profile_name = datastream_connection_profile.my_profile.name
  
  destination_config {
    cloud_storage_source_config {...}
  }
  backfill {...}
  source_config {
    mysql_source_config {...}
  }
}

resource "datastream_connection_profile" "my-profile" {
  name = "..."
  display_name = "My Profile"

  mysql_profile {...}
  private_connectivity {
    private_connection_name = datastream_private_connection.my_private_connection
  }
}

resource "datastream_private_connection" "my_private_connection" {
  name = "..."
  display_name = "My Private Connection"
  
  vpc_peering_config {
    subnet = ""
    vpc = ""
  }
}

References

This might be a helpful starting point: https://cloud.google.com/datastream/docs/use-the-cloud-sdk

[joshtree41]

any update on when this might get some love?

[red8888]

Bump. Where is this on the roadmap? Datastream is "just a few clicks", but I want to reign in this config with IaC.

[arunsingh2803]

I have written a python based automation for end to end deployment of DataStream. You can use this as IaC or integrate it with existing IaC for DataStream till terraform module for DataStream comes
https://github.com/GoogleCloudPlatform/professional-services/tree/main/tools/datastream-deployment-python-automation

[trentkgUjet]

I would love this feature!

[spideyz0r]

Waiting for it!!

[joshtree41]

any update?

[trentkgUjet]

I'm going to give this feature a try myself.

[jlowe64]

I have done a bit of work on a terraform module using @arunsingh2803 's script posted above. I have ripped some of his features out and added some features of my own. Would anyone be interested in me publishing this and allowing you all to modify it to be a fully featured module?

[melinath]

b/233804820

[melinath]

@trentkgUjet @jlowe64 thanks for offering to contribute - just wanted to give you a heads up I've started working on this ticket from the core contributor side. Apologies if you've already started work on this! Just found the Github ticket when I made my first PR :-)

[trentkgUjet]

@melinath Oh excellent! Do you have anything to show yet? I have only gotten as far as familiarizing myself with the work needed. I've setup the environment for magic-modules and can build and compile the go code. Happy to yield this to someone with more experience.

[melinath]

I'll reference this ticket in all PRs so they'll show up in the comment thread as I make them :-) The first one is above my previous comment.

[Rstar1998]

Any updates on when we will be able to create datastream resources using terraform ? Have to created everything manually 😞

[equals215]

@melinath beautiful! If you ever need help on producing some code please ping me, I'd be more than happy to contribute to the gcp provider

[laksh-bhat]

Hey @melinath we are so happy this is now in terraform and can't wait for the stream resource too :)

We've been running into issues when destroying private connections resource "google_datastream_private_connection" "datastream_conn" {} with the error

│ Error: Error when reading or editing PrivateConnection: googleapi: Error 400: Resource '"projects/internal/locations/us-central1/privateConnections/datastream-conn"' has nested resources. If the API supports cascading delete, set 'force' to true to delete it and its nested resources.

this resource may need to support the equivalent of --force flag

[melinath]

@laksh-bhat could you please open a new bug issue & fill out the fields there? Thanks!

[obrienciaran]

Definitely looking forward to the stream resource feature too!

[juleslenotte]

Still highly interested in having the datastream resources managed in terraform over here.

[melinath]

Some work remaining on this.

[GusPrice]

Super exciting to have this resource available! I saw in your PRs:

In particular I'm excluding CMEK fields and oracle & postgres (because those require specific handwritten tests) and oracle & postgres support (because that requires separate, non-automated testing.)

Is there an intent to add postgres support as a follow up?

[fryck]

Perfect !

I have an error when I try to update an existing stream:

Error: Error waiting for Updating Stream: {"@type":"type.googleapis.com/google.rpc.ErrorInfo","domain":"datastream.googleapis.com","metadata":{"message":"Some validations failed to complete successfully, see the full list in the operation metadata.","originalMessage":"","time":"2023-02-01T14:09:20Z","uuid":"05945f93-a23a-11ed-a1af-b2d514f7a161"},"reason":"VALIDATION_FAILURE"}

Thanks !

[roaks3]

The last PR for this work was merged yesterday, which includes Postgres and Oracle sources: GoogleCloudPlatform/magic-modules#7175.

Marking this issue as closed as support for these resources has been added, but for those experiencing an issue or potential bug, please open a new issue so that we can track it separately.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.