[New Resource]: S3tables catalog creation in Glue Data Catalog

[alfonsozamorac]

Description

For the creation of the catalog in Glue Data Catalog for S3 Tables, it is necessary to run a command to enable it. If the resource aws_glue_catalog_database is used, it creates a database but not a catalog, and it doesn't link with LakeFormation Catalog (new feature) as it does when running the aws glue create-catalog command.

Requested Resource(s) and/or Data Source(s)

aws_glue_catalog or aws_lakeformation_catalog

Potential Terraform Configuration

aws glue create-catalog \
--region us-east-1 \
--cli-input-json file://catalog.json


catalog.json:


{
   "Name": "s3tablescatalog",
   "CatalogInput": {
      "FederatedCatalog": {
          "Identifier": "arn:aws:s3tables:us-east-1:111122223333:bucket/*",
          "ConnectionName": "aws:s3tables"
       },
       "CreateDatabaseDefaultPermissions":[],
       "CreateTableDefaultPermissions":[]
   }
}

References

https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[CyrusTC]

Additionally, consider the create-catalog command for other source like redshift as well:

aws glue create-catalog \
        --cli-input-json '{
          "Name": "aurora-lakehouse-catalog",
          "CatalogInput": {
            "Description": "Aurora data via Redshift managed storage",
            "CatalogProperties": {
              "DataLakeAccessProperties": {
                "DataLakeAccess": "true",
                "DataTransferRole": "${aws_iam_role.data_transfer.arn}",
                "CatalogType": "aws:redshift"
              }
            }
          }
        }

[YakDriver]

Just to clarify the relationship between related S3Tables issues:

• [Bug]: LakeFormation permissions for S3Tables Catalog #40724 addresses LakeFormation permissions validation for existing S3Tables catalogs (the catalog_id format validation bug)
• [New Resource]: S3tables catalog creation in Glue Data Catalog  #40725 (this issue) requests a new resource for creating federated catalogs via aws glue create-catalog

@CyrusTC's point about Redshift catalog support is spot-on - the same aws_glue_federated_catalog resource should handle both S3Tables and Redshift sources.

Related Glue/S3Tables issues worth considering together:

• Glue Ray jobs fail to update #35012/[Bug]: Ray Glue jobs fail to update #35013 - Glue Ray job update issues (may share catalog interaction patterns)
• [Bug]: Error when calling resource "aws_glue_catalog_table" An error occurred (EntityNotFoundException) when calling the GetPartitionIndexes operation: Table image_metadata not found. #34132 - Glue catalog table EntityNotFoundException (catalog existence validation)
• r/aws_glue_job: Allow a zero (0) value for timeout for Apache Spark streaming ETL jobs #44920 - Glue job timeout configuration (catalog operation timeouts)

For maintainers reviewing this: The AWS CLI example shows the federated catalog structure clearly - this would likely be implemented as aws_glue_federated_catalog (such as #43583) with a federated_catalog block supporting both s3tables and redshift connection types. The CreateDatabaseDefaultPermissions and CreateTableDefaultPermissions arrays are key for LakeFormation integration.

This catalog creation piece would complete the foundational S3Tables + Glue + LakeFormation workflow.

[cklat]

Is this resource available in the awscc provider for the time being?

[elventear]

Seems you can't create Glue catalogs with CF.

[ztai-add]

I have the same issue that s3tablescatalog isn't created until s3 tables integration with analytic services is enabled, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-tables-integrating-aws.html

[andreafalzetti]

Same issue