feat(ingestion/SageMaker): Remove deprecated apis and add stateful ingestion capability

[TonyOuyangGit]

This PR has the following improvements:

1. remove deprecated APIs: Amazon Sagemaker Edge retired on April 26th, 2024, so the calls to any edge packaging-related jobs will fail with errors: https://docs.aws.amazon.com/sagemaker/latest/dg/edge-eol.html, we should clean up all edge packaging related code
2. according to https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_DescribeAutoMLJobV2.html, DescribeAutoMLJobV2 are the new versions of CreateAutoMLJob and DescribeAutoMLJob which offer backward compatibility, the PR updates this API call to use this newer API
3. adds stateful ingestion capability to SageMaker ingestion

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[hsheth2]

one small thing, otherwise looks good

[hsheth2]

it doesn't look like this source supports platform instances, so having it in the config might be misleading

[TonyOuyangGit]

Thank you! It's interesting that if I give platform_instance in the ingestion recipe, then I can see the platform instance available when browsing ML models, it's in the response of graphQL query getBrowseResultsV2 even though each ML models entity does not have platform_instance.

Do you think I can make some more improvements to support platform instances? Most of the make_ml_model..*_urn code is in mce_builder.py.

[hsheth2]

yeah adding support for it isn't too bad. it involves

1. adding platform instance to the config (you already did this)
2. adding the platform instance as a prefix to all urns. modifying the helpers in mce_builder.py to take an optional platform_instance is the right approach there
3. emit dataPlatformInstance aspects for every entity

the browse path v2 aspects will get auto-generated based on everything else