[Feature Store] Feature group creation: provide a DataCatalogConfig while enabling glue table creation

[simonvdk]

Confirm by changing [ ] to [x] below:

• I've gone though the User Guide and the API reference
• I've searched for previous similar issues and didn't find any solution

Issue is about usage on:

• Service API : I want to do X using Y service, what should I do?
• CLI : passing arguments or cli configurations.
• Other/Not sure.

Platform/OS/Hardware/Device
Linux

Describe the question
Use case
Create a feature group with automatic glue table creation for the offline store metadata, while configuring the glue data catalog database and table names

Issue encountered
It seems that providing a DataCatalogConfig and setting disable_glue_table_creation to false are mutually exclusive:

• I can either not configure the glue database and table names and enable the glue table creation, so that the glue table with the default name and database is created upon feature group creation
• OR I can provide a DataCatalogConfig but then I have to disable the glue table creation, so that the requested glue table is not created upon feature group creation

But I cannot provide a DataCatalogConfig and enable the glue table creation. Error encountered:

An error occurred (ValidationException) when calling the CreateFeatureGroup operation: Validation Error: DataCatalogConfig is not permitted in the request unless AutoCreateGlueTable is turned off. Please either set AutoCreateGlueTable to false or remove DataCatalogConfig from the request.

Why this seems to be an issue:

• this behaviour (mutually exclusive) is not mentioned in the documentation. Also, there is no further mention or example of how to configure the offline store data catalog in the documentation
• given the current state of the documentation, a user may want to configure the name of the glue database and table where the offline store metadata will be stored, while benefiting from the glue table creation upon feature group creation (with all the configuration - schema, storage descriptor etc - coming from the feature group information)
• this extract from the java SDK documentation seems to indicate that the DataCatalogConfig should not be mutually exclusive with the automatic table creation

Ways to reproduce issue
Reproduced with AWS SDK (2.50.0) and AWS CLI.
Providing an OfflineStoreConfig with both DisableGlueTableCreation=False and a DataCatalogConfig with configured glue database (already created) and a glue table (that does not yet exist) raises the above error. Proving the DataCatalogConfig with DisableGlueTableCreation=True does not raise, but the glue table is not created either.

Example with AWS CLI:

aws sagemaker create-feature-group --cli-input-json '{"EventTimeFeatureName": "timestamp", "Description": "", "RecordIdentifierFeatureName": "record_id", "FeatureDefinitions": [{"FeatureName": "record_id", "FeatureType": "Integral"}, {"FeatureName": "timestamp", "FeatureType": "String"}], "OfflineStoreConfig": {"S3StorageConfig": {"S3Uri": "s3://my_bucket/my_prefix", "KmsKeyId": "arn:aws:kms:region:account_id:key/key_id"}, "DataCatalogConfig": {"TableName": "my_table", "Catalog": "account_id", "Database": "my_db"}, "DisableGlueTableCreation": false}, "FeatureGroupName": "my-feature-group"}'

Expected output
A clearer documentation about how to configure the offline store data catalog (e.g. with an example in a notebook), and possibly the possibility to configure the data catalog while benefiting from the glue table creation

Logs/output
Get full traceback and error logs by adding --debug to the command.

NB: A similar issue has been opened on the sagemaker-python-sdk repository

[kdaily]

Hi @simonvdk,

Thanks for your post. The error you're receiving is from the server, so this functionality is not currently available. This looks like a limitation of the API, not of the AWS CLI or AWS SDKs. The documentation within the Java SDK you're pointing to comes directly from the API documentation. The documentation for the command you referenced (aws sagemaker create-feature-group) says:

To learn more about this parameter, see OfflineStoreConfig .

but it does not provide a link to where this documentation is - it's referring to the API documentation linked above.

The Sagemaker Developer guide does have notebooks with examples and more information about feature stores - I don't see anything specifically discussing your use case.

• https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store.html
• https://docs.aws.amazon.com/sagemaker/latest/dg/feature-store-notebooks.html

I will pass along your feedback to the Sagemaker team regarding the functionality and the documentation. The Sagemaker team also manages the repository that you cross-posted to, so you may get a more direct response there about the use case you've described.

Another option to get feedback on your use case would be to ask on repost.aws.

However, since this is not an issue directly with the functionality of the AWS CLI, I'm going to close this issue. If I get any feedback from the Sagemaker team, I will share it here. I hope this provides you some options to go forward.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[clausagerskov]

so is this supported or not?