WIP: Data catalog #2

tumido · 2020-10-22T15:08:04Z

💥 💣 This is WIP. Do not merge. 💣 💥

I've worked on kustomize-ing the Data Catalog for Internal DH on: AICoE/idh-manifests#29, AICoE/idh-manifests#27

This PR is an attempt to convert those manifests to a scheme consumable via kfdef. This is a custom, additional, component to ODH and the aim is to have it implemented here and usable for operate-first, before it lands in ODH upstream (that can take time, but we need to have it usable right now.

To-be-done:

Debug the manifests
Ensure credentials exchange and connections between Hue, Thriftserver and Hive and their databases
Assess database unification, or usage of a provided service via operator, instead of a custom DM manifests for mysql and postgresql.
Check against https://gitlab.com/opendatahub/opendatahub-operator/-/tree/master/roles/

sesheta · 2020-10-27T16:09:21Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign tumido after the PR has been reviewed.
You can assign the PR to them by writing /assign @tumido in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sesheta · 2020-10-27T16:11:42Z

Pre-Commit Test failed! Click here

[INFO] Initializing environment for git://github.com/Lucas-C/pre-commit-hooks.
[INFO] Initializing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/adrienverge/yamllint.git.
[INFO] Installing environment for git://github.com/Lucas-C/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/adrienverge/yamllint.git.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Tabs remover.............................................................Passed
Trim Trailing Whitespace.................................................Passed
Check for merge conflicts................................................Passed
Fix End of Files.........................................................Passed
Check for added large files..............................................Passed
Check for case conflicts.................................................Passed
Check JSON...........................................(no files to check)Skipped
Check for broken symlinks............................(no files to check)Skipped
Detect Private Key.......................................................Passed
Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Passed
yamllint.................................................................Failed
- hook id: yamllint
- exit code: 2

kfdef/datacatalog/manifests/base/thriftserver/postgres-dc.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/hue/hue-mysql-secret.yaml
  1:1       warning  missing document start "---"  (document-start)
  9:81      error    line too long (83 > 80 characters)  (line-length)

kfdef/datacatalog/manifests/base/hue/hue-service.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/thriftserver/postgres-secret.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/hue/hue-route.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/thriftserver/postgres-service.yaml
  1:1       warning  missing document start "---"  (document-start)
  6:81      error    line too long (109 > 80 characters)  (line-length)

kfdef/datacatalog/manifests/base/hue/kustomization.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/thriftserver/thriftserver-route.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/hue/hue-dc.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/superset/kustomization.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/thriftserver/kustomization.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/thriftserver/thriftserver-hdfs-hive-secret.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/kfdef.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/hue/hue-mysqld-exporter-secret.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/thriftserver/thriftserver-dc.yaml
  1:1       warning  missing document start "---"  (document-start)

operator/base/operator_group.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/hue/hive-site-xml-secret.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/configurations.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/thriftserver/thriftserver-pvc.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/hue/hue-mysql-pvc.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/hue/hue-mysql-service.yaml
  1:1       warning  missing document start "---"  (document-start)
  6:81      error    line too long (103 > 80 characters)  (line-length)

operator/base/subscription.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/thriftserver/postgres-exporter.yaml
  1:1       warning  missing document start "---"  (document-start)
  10:81     error    line too long (101 > 80 characters)  (line-length)

kfdef/datacatalog/manifests/base/hue/hue-mysql-dc.yaml
  1:1       warning  missing document start "---"  (document-start)

.aicoe-ci.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/thriftserver/postgres-pvc.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/kustomization.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/jupyterhub/kustomization.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/jupyterhub/kfdef.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/hue/hue-db-init-job.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/kustomization.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/superset/kfdef.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/thriftserver/thriftserver-service.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/manifests/base/thriftserver/thriftserver-server-conf-secret.yaml
  1:1       warning  missing document start "---"  (document-start)
  7:81      error    line too long (94 > 80 characters)  (line-length)
  14:81     error    line too long (121 > 80 characters)  (line-length)

kfdef/datacatalog/manifests/base/hue/hue-ini-secret.yaml
  1:1       warning  missing document start "---"  (document-start)

operator/base/kustomization.yaml
  1:1       warning  missing document start "---"  (document-start)
  5:1       error    wrong indentation: expected 2 but found 0  (indentation)

sesheta · 2020-11-03T14:21:58Z

Pre-Commit Test failed! Click here

[INFO] Initializing environment for git://github.com/Lucas-C/pre-commit-hooks.
[INFO] Initializing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/adrienverge/yamllint.git.
[INFO] Installing environment for git://github.com/Lucas-C/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/adrienverge/yamllint.git.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Tabs remover.............................................................Passed
Trim Trailing Whitespace.................................................Passed
Check for merge conflicts................................................Passed
Fix End of Files.........................................................Passed
Check for added large files..............................................Passed
Check for case conflicts.................................................Passed
Check JSON...........................................(no files to check)Skipped
Check for broken symlinks............................(no files to check)Skipped
Detect Private Key.......................................................Passed
Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Passed
yamllint.................................................................Failed
- hook id: yamllint
- exit code: 2

kfdef/superset/kustomization.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/kfdef.yaml
  1:1       warning  missing document start "---"  (document-start)

operator/base/operator_group.yaml
  1:1       warning  missing document start "---"  (document-start)

operator/base/subscription.yaml
  1:1       warning  missing document start "---"  (document-start)

.aicoe-ci.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/datacatalog/kustomization.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/jupyterhub/kustomization.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/jupyterhub/kfdef.yaml
  1:1       warning  missing document start "---"  (document-start)

kfdef/superset/kfdef.yaml
  1:1       warning  missing document start "---"  (document-start)

operator/base/kustomization.yaml
  1:1       warning  missing document start "---"  (document-start)
  5:1       error    wrong indentation: expected 2 but found 0  (indentation)

tumido · 2020-11-03T16:42:53Z

cross-linking upstream ODH PRs opendatahub-io/odh-manifests#227 opendatahub-io/odh-manifests#228

tumido · 2020-11-03T16:50:55Z

This will be probably closed in favor of upstream ODH manifests. Also a replacement/additional PR will be landed here - with encrypted overlays for MOC deployment.

tumido · 2020-11-24T12:01:12Z

Upstream datacatalog is working and ready to be deployed. Closing in favor of kfdef resource using the upstream.

sesheta added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 22, 2020

tumido mentioned this pull request Oct 23, 2020

Docs for overrides #3

Merged

HumairAK added this to In progress in Master Board Oct 27, 2020

tumido added 6 commits October 27, 2020 17:03

cherry-pick: Split hue base to manageable yaml files

7a3f211

cherry-pick: Fix selectors for Services

4c5862c

cherry-pick: Fix s/name/secretName/ in volumes

58d8255

cherry-pick: Add Thriftserver base

06ce7b1

Reorganize data catalog base

e7bcc17

Changes to data-catalog

eb58fd2

tumido force-pushed the data-catalog branch from 29e10b1 to 936d496 Compare October 27, 2020 16:09

Restructure after operate-first/odh repo layout changes

936d496

Dont template db secrets, flagged for future removal

dfb0e32

durandom mentioned this pull request Nov 5, 2020

Add Data catalog from internal DH #15

Closed

8 tasks

HumairAK moved this from In progress to In Review in Master Board Nov 12, 2020

HumairAK moved this from In Review to In progress in Master Board Nov 12, 2020

tumido closed this Nov 24, 2020

Master Board automation moved this from In progress to November 2020 Nov 24, 2020

tumido deleted the data-catalog branch November 2, 2022 10:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Data catalog #2

WIP: Data catalog #2

tumido commented Oct 22, 2020 •

edited

Loading

sesheta commented Oct 27, 2020

sesheta commented Oct 27, 2020

sesheta commented Nov 3, 2020

tumido commented Nov 3, 2020

tumido commented Nov 3, 2020

tumido commented Nov 24, 2020

WIP: Data catalog #2

WIP: Data catalog #2

Conversation

tumido commented Oct 22, 2020 • edited Loading

sesheta commented Oct 27, 2020

sesheta commented Oct 27, 2020

sesheta commented Nov 3, 2020

tumido commented Nov 3, 2020

tumido commented Nov 3, 2020

tumido commented Nov 24, 2020

tumido commented Oct 22, 2020 •

edited

Loading