[Enhancement]: Support Snowflake type for aws_glue_connection

[sreid]

Description

Request for adding support for the (in preview) Snowflake connection type for AWS Glue. This allows for Glue to make managed connections to Snowflake instances. See references section for links to announcement and documentation.

Note: I'm unsure when/how the provider chooses to implement things that are in "Preview", and realize it may need to wait until a GA release. I can confirm that I've used the functionality via the AWS console, and it worked as expected.

Affected Resource(s) and/or Data Source(s)

• aws_glue_connection

Potential Terraform Configuration

resource "aws_glue_connection" "snowflake-connection" {
  name = "snowflake-connection"
  connection_type = "SNOWFLAKE"

  connection_properties = {
    SNOWFLAKE_URL  = "https://your_instance.us-east-1.snowflakecomputing.com"
    SNOWFLAKE_ROLE = "OPTIONAL_ROLE_NAME"
    SECRET_ID      = "configured-secret-name"
  }
  ...
}

References

• https://aws.amazon.com/about-aws/whats-new/2023/06/aws-glue-native-snowflake-connectivity-etl-capabilities-preview/
• https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-snowflake-home.html

Would you like to implement a fix?

No

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[BrettSGA]

In this article they're using jdbc:snowflake://<snowflake account info> which may work within TF as:

resource "aws_glue_connection" "example" {
  connection_properties = {
    JDBC_CONNECTION_URL = "jdbc:snowflake://<snowflake url>"
    PASSWORD            = "examplepassword"
    USERNAME            = "exampleusername"
  }
  name = "example"
}

Haven't gotten a chance to try it out myself but I'll update when I do.

[DrFaust92]

According to glue sdk docs, there is no such option. but it seems the correct way to do it is how @BrettSGA suggested above.

Closing for now, please re-open with relevant details

[elbud]

There is definitely a SNOWFLAKE type connector available in preview for ETL jobs. I also can't find it anyway in the SDK but it's available through the console so must be somewhere.

https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html

[sreid]

@elbud correct.

If I run a aws glue get-connection to CLI command to describe the connection, a working setup (created interactively in the console) looks something like this:

{
    "Connection": {
        "Name": "my-snowflake-connection",
        "Description": "example snowflake setup",
        "ConnectionType": "SNOWFLAKE",
        "ConnectionProperties": {
            "SparkProperties": "{\"sfUrl\":\"https://example.us-east-1.snowflakecomputing.com\",\"secretId\":\"my-snowflake-for-glue-secret\",\"sfRole\":\"MY_ROLE\"}"
        },
        "PhysicalConnectionRequirements": {
            "SubnetId": "[...]",
            "SecurityGroupIdList": [
                "[...]"
            ],
            "AvailabilityZone": "us-west-2a"
        },
        "CreationTime": "2023-08-17T12:10:44.491000-07:00",
        "LastUpdatedTime": "2023-08-19T15:34:26.455000-07:00"
    }
}

@DrFaust92 correct me if wrong, but I think this is different? (agreed using JBDC is a possible way, although more complicated than the newer option in preview)

[mbainter]

This was only announced 6/26/23, and it seems like some docs aren't yet updated. the aws-cli can even work with the SNOWFLAKE type using the input param.

@DrFaust92 You should re-open this.

[bhatla1983]

Tried creating a connection/job using "SNOWFLAKE" as connector, this is so much better. Next step for me is to convert this to code, eagerly waiting for this option in Terraform...

[DrFaust92]

setting as upstream as this is not yet supported in go sdk

[wdfinch]

I also would really like to see this. I tried using the JDBC method and couldn't get it to work. Additionally, the glueContext object in the SDK seems to only pull from Snowflake when this type of connection is used.

[FelixGagnonB]

Hello,
We would also really like this feature. We have to interact with Snowflake and for now, we are manually creating the connection instead of using Terraform.

Thanks

[cristianrat]

Also in need of this

[MyJBMe]

Just used the connector for the first time. Coming from the old way of connecting this is much better. To make it perfect, only terraform support is missing!

[DrFaust92]

API is still not updated /: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-connections.html#aws-glue-api-catalog-connections-ConnectionInput

[cristianrat]

To anyone coming to this thread now - a solution can be to write a small CloudFormation template for the connector, and use terraform to apply it. You can still put in your values etc (ie: string interpolation).
I ended up doing that, works well, even if I'm no fan of CloudFormation

[DrFaust92]

We ca potentially remove the validation but 1. it break validation 2. there isnt a properly documented list of valid values /:

[Emanuel071]

is this the official way to request terraform support to add the resource "aws_glue_connection" ?

[Emanuel071]

In this article they're using jdbc:snowflake://<snowflake account info> which may work within TF as:

resource "aws_glue_connection" "example" {
  connection_properties = {
    JDBC_CONNECTION_URL = "jdbc:snowflake://<snowflake url>"
    PASSWORD            = "examplepassword"
    USERNAME            = "exampleusername"
  }
  name = "example"
}

Haven't gotten a chance to try it out myself but I'll update when I do.

have you found success in this ?