Feature: Add wait with logs to subset of Job types

[benieric]

Issue #, if available:

Description of changes:

• Add log stream handlers that will be used for streaming cloudwatch log events during wait

  • MultiLogStreamHandler - used for handling cases where we need to stream events from multiple streams in the same log group
  • LogStreamHandler - single log stream handler

• Only adding support for a subset of job type: set(["TrainingJob", "ProcessingJob", "TransformJob"])

  • this is only that sagemaker v2 supports today, but is extensible for future job types

• Both MultiLogStreamHandler and LogStreamHandler utilize a get_latest_log_event() method which returns a generator that yields:

str: stream_name,
dict: event dict in format
    { 
        "ingestionTime": number,
        "message": "string",
        "timestamp": number
    }

• Fix rich logging formatting as well - Duplicate Logging Info with Rich logger #172

Testing

• Unit tests
• Testing in notebook
  

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[nargokul]

Nit : Not a sagemaker client

[nargokul]

If it is really needed only once , we can create it during the first call and have a check

if not self.client: 
      session = Session()
      self.client = session.client(
          "logs",
          session.region_name,
          config=Config(retries={"max_attempts": 10, "mode": "standard"}),
      )

[benieric]

Yes, let me add this, good catch