DB: Add environment variable option to skip DB table creationˆ

[lkaybob]

What this PR does / why we need it:

Currently, DB Manager automatically tries to create observation_logs table on startup using CREATE TABLE IF NOT EXISTS clause. However, this part does not work when DB user does not have enough privilege to create table if DB schema creation is handled by manual admin process.

To address this, we have hard-forked the repository and built our own image after commenting out table creation part so far. Since I guess there might be similar need to skip table creation from other than us, I've added an environment variable option to skip table creation along with simple logic to validate observation_logs table structure with SELECT statement.

(ml-metadata allows this with option in protobuf text format.)

Which issue(s) this PR fixes

None. I've create pull request directly.

Checklist:

• Docs included if any changes are user facing

(Need to update list of Environment Variables for Katib Components on Katib DB Manager. I will create additional pull request to address the change on website.)

FYI. I've created an issue for Pipelines(kubeflow/pipelines#10265) where similar improvement could be possible

[andreyvelich]

Thank you for creating this @lkaybob!
I agree, we should give user capability to skip DB creation.
I left a few comments.
/assign @tenzen-y @johnugeorge

[tenzen-y]

@lkaybob Thank you for this contribution :)
Totally, LGTM: I left a few comments.

[lkaybob]

Thanks for the review! @andreyvelich @tenzen-y
I've applied suggestions you left with additional changes to make consistent.

[tenzen-y]

It looks great to me.
@kubeflow/wg-automl-leads Can you approve workflows?

/lgtm
/approve

[tenzen-y]

/lgtm cancel
(We need to wait for succeeded workflows)

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lkaybob, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[andreyvelich]

Thank you for the updates @lkaybob!
Please can you create an issue or website PR to track the documentation updates that are required for Katib

[tenzen-y]

@kubeflow/wg-automl-leads Could you restart CI jobs, Go Test / Unit Test (1.26.1) and Go Test / Unit Test (1.27.1)?

[andreyvelich]

It's strange why trial_controller_test constantly fails.
@lkaybob Please can you try to rebase your PR ?

[lkaybob]

@andreyvelich I've just rebased my PR, and I will create an issue at kubeflow/website repository to address this change so that I can work on it later.

[tenzen-y]

This is blocked by #2251

[andreyvelich]

@lkaybob Please can you rebase one more time ?
It should fix the Python test.

[lkaybob]

@andreyvelich I've just rebased the branch. Let's see if the test passes :)

[andreyvelich]

Thank you for this @lkaybob!
/lgtm