Query Athena in the Pipeline Object with Parameters

[idanmoradarthas]

Describe the feature you'd like
As of today, using the AthenaDatasetDefinition object as an input in ProcessingInput, require a string SQL query. If the query is dependent on pipeline parameters (such as dates) you are unable to use it. Placing the parameters in string converts them automatically into their default value in the upsert stage, and thus when executing the pipeline with different parameters creates the wrong query.

How would this feature be used? Please describe.
The query_string attribute in the AthenaDatasetDefinition object will be able to calculate pipeline parameters, as output_s3_uri is able with the object sagemaker.workflow.functions.Join

[idanmoradarthas]

Try to do it with sagemaker.workflow.functions.Join got the following error:
Traceback (most recent call last):
File "/root/.pyenv/versions/3.9.12/lib/python3.9/site-packages/sagemaker/workflow/pipeline.py", line 180, in upsert
response = self.create(role_arn, description, tags)
File "/root/.pyenv/versions/3.9.12/lib/python3.9/site-packages/sagemaker/workflow/pipeline.py", line 116, in create
return self.sagemaker_session.sagemaker_client.create_pipeline(**kwargs)
File "/root/.pyenv/versions/3.9.12/lib/python3.9/site-packages/botocore/client.py", line 415, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/root/.pyenv/versions/3.9.12/lib/python3.9/site-packages/botocore/client.py", line 745, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the CreatePipeline operation: Unable to parse pipeline definition. Field ProcessingInputs.DatasetDefinition.AthenaDatasetDefinition.QueryString must be a literal value, not a function.
Exception: An error occurred (ValidationException) when calling the CreatePipeline operation: Unable to parse pipeline definition. Field ProcessingInputs.DatasetDefinition.AthenaDatasetDefinition.QueryString must be a literal value, not a function.

[ShwetaSingh801]

Hi @idanmoradarthas,

Thanks for using SageMaker and taking the time to suggest ways to improve SageMaker Python SDK. We have added your feature request it to our backlog of feature requests and may consider putting it into future SDK versions. I will go ahead and close the issue now, please let me know if you have any more feedback. Let me know if you have any other questions.

Best,
Shweta