chore: End-to-End Tabular System Test

[vinnysenthil]

Summary of Changes

• Add a end-to-end test that covers Custom Training and AutoML Tabular CUJ, including:
  • Dataset creation and import
  • Creating and running CustomTrainingJob with inner script and AutoMLTabularTrainingJob
  • Creating and deploying models to two Endpoints
  • Online prediction to both models and checking rough accuracy
• The addition of another presubmit config to execute nox session system-3.8 for pull requests like this one (editing of system tests)

Fixes b/172263198 🦕

[sasha-gitg]

It seems like it would be more reusable to create a base class that includes all these fixtures, if pytest support this. It would make creating new tests avoid this boilerplate.

Separating the creation of the bucket and uploading of the test dataset would make it easier to reuse.

[sasha-gitg]

What are the advantages of creating a new bucket for each integration test vs using a common bucket and creating a new directory?

[vinnysenthil]

@sasha-gitg I split out re-usable logic to a base TestEndToEnd class, lmk if you think we should structure it differently. We can have further discussion offline on the shared_state pattern.

One last TODO before merging is to have Presubmit - System Tests CI to trigger on this PR

[sasha-gitg]

Thanks Vinny, this looks great! Left a few minor comments.