IngestionManagerPandas doesn't propagate Boto3 Session profile_name to Workers

[apriceaws]

Describe the bug
When ingesting a DataFrame to a FeatureGroup, there is no way to set the profile_name in the Boto3 Session used by IngestionManagerPandas workers/threads. This prevents utilizing anything but default credentials when ingesting to a FeatureGroup using this class.

To reproduce

1. Ensure "default" creds profile does not have access to write to FeatureGroup
2. Ensure "<profile_name>" creds profile does have access to write to FeatureGroup
3. Set a profile_name on boto3 using boto3.setup_default_session(profile_name=<profile_name>)
4. Attempt to ingest a dataframe to a FeatureGroup using .ingest()
5. Ingest will fail because "default" creds do not have access, and "<profile_name>" creds will not be used in the workers

Expected behavior
Worker Boto3 sessions should be initialized with a profile name which matches boto3.DEFAULT_SESSION.profile_name.

System information
A description of your system. Please provide:

• SageMaker 2.38.0

Additional context
Use of default credentials is not an option in this context.

[apriceaws]

I believe this is related to #2288.

[apriceaws]

Confirmed that this worked fine in SageMaker 2.37

[ahsan-z-khan]

@alex-tang , can you take a look at this?

[gustavovaliati]

Any news on that? We still have got the same issue at v2.51.

[kusumbhatt]

@apriceaws Apologies for getting back to this so late.

Earlier the profile name was not trickled down to the worker sessions in the ingest(..) method and hence the workers were making use of default credentials.

This now supports providing "profile_name" as an argument in ingest(..) method and the worker sessions are initialized with the profile parameter. So, in order to use the profile_name different than default, it should be passed to profile_name argument in the ingest(..) method. In case, the profile_name argument is not specified, the ingest(..) method uses the default credentials.

Can you please confirm if this helps you?

[zhaoqizqwang]

Closing this ticket since the profile_name is supported as an argument in newer sagemaker