Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data Catalog - Thriftserver #29

Merged
merged 8 commits into from
Nov 2, 2020
Merged

Conversation

tumido
Copy link
Member

@tumido tumido commented Oct 22, 2020

Related to: #27 (either of them will need a rebase, if the other is merged)
Part of: DATAHUB-2295

This converts the Thriftserver component to kustomize base.

The layout is the same as for the Hue component in #27

  1. In data-catalog/base/thriftserver are all the manifests needed for Thriftserver
  2. The kustomization file in base, data-catalog/base/thriftserver/kustomization.yaml, allows to specify image version + adds a app=thriftserver label
  3. For each env we overlay the base - modifying all the secrets, sops encrypted.
  4. ConfigMaps which contained sensitive data - (like db passwords) are transformed to Secrets and mounted in the DeploymentConfigs accordingly
  5. Overlays replaces the secret config file values - setting cores/memory limits, credentials, target namespaces as well as merging the storage-class on PVCs

@sesheta sesheta added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 22, 2020
@tumido
Copy link
Member Author

tumido commented Oct 22, 2020

cc @maulikjs @4n4nd, @accorvin please review. I'm sure I made mistake somewhere, especially in those secrets...

@sesheta
Copy link

sesheta commented Oct 22, 2020

Pre-Commit Test failed! Click here
[INFO] Initializing environment for git://github.com/Lucas-C/pre-commit-hooks.
[INFO] Initializing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Installing environment for git://github.com/Lucas-C/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Tabs remover.............................................................Passed
Trim Trailing Whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing jupyterhub/bases/custom-images/jupyterhub-custom-images.yaml
Fixing kafka/README.md

Check for merge conflicts................................................Passed
Fix End of Files.........................................................Passed
Check for added large files..............................................Passed
Check for case conflicts.................................................Passed
Check JSON...........................................(no files to check)Skipped
Check for broken symlinks............................(no files to check)Skipped
Detect Private Key.......................................................Passed
Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Passed

@maulikjs
Copy link

Tested this out and works great! Thank you sooooo much @tumido, we are good to merge this in!

@maulikjs
Copy link

/approve

@maulikjs
Copy link

/assign @HumairAK

@tumido
Copy link
Member Author

tumido commented Oct 28, 2020

Btw, both of those PR's are against the data catalog branch. Once both are merged we'd need to merge it all back to production.

@maulikjs
Copy link

Yup Yup @tumido
Once merged into data-catalog I will test them once more against all our environments and have that merged into production.
I can drive that effort.

@tumido tumido mentioned this pull request Oct 30, 2020
@sesheta
Copy link

sesheta commented Nov 2, 2020

Pre-Commit Test failed! Click here
[INFO] Initializing environment for git://github.com/Lucas-C/pre-commit-hooks.
[INFO] Initializing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Initializing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Installing environment for git://github.com/Lucas-C/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for git://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pre-commit/pre-commit-hooks.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
Tabs remover.............................................................Passed
Trim Trailing Whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

Fixing jupyterhub/bases/custom-images/jupyterhub-custom-images.yaml
Fixing kafka/README.md

Check for merge conflicts................................................Passed
Fix End of Files.........................................................Passed
Check for added large files..............................................Passed
Check for case conflicts.................................................Passed
Check JSON...........................................(no files to check)Skipped
Check for broken symlinks............................(no files to check)Skipped
Detect Private Key.......................................................Passed
Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Passed

@maulikjs
Copy link

maulikjs commented Nov 2, 2020

Thanks for the changes @tumido, works now!

@maulikjs
Copy link

maulikjs commented Nov 2, 2020

/approve

1 similar comment
@anishasthana
Copy link

/approve

@sesheta
Copy link

sesheta commented Nov 2, 2020

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anishasthana, maulikjs

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sesheta sesheta added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 2, 2020
@sesheta sesheta merged commit 0e74266 into AICoE:data-catalog Nov 2, 2020
@maulikjs maulikjs mentioned this pull request Nov 2, 2020
maulikjs added a commit that referenced this pull request Nov 2, 2020
* Add Thriftserver base

* Add dev overlays

* Add stage overlays

* Add prod-* overlays

* Cleanup: remove vars

* Enlist thriftserver overlays

* Patch postgres dc for ImageChange trigger

Co-authored-by: Maulik Shah <maulik.shah1995@gmail.com>
maulikjs added a commit that referenced this pull request Nov 2, 2020
* Add Thriftserver base

* Add dev overlays

* Add stage overlays

* Add prod-* overlays

* Cleanup: remove vars

* Enlist thriftserver overlays

* Patch postgres dc for ImageChange trigger

Co-authored-by: Maulik Shah <maulik.shah1995@gmail.com>
maulikjs added a commit that referenced this pull request Nov 3, 2020
* Add Thriftserver base

* Add dev overlays

* Add stage overlays

* Add prod-* overlays

* Cleanup: remove vars

* Enlist thriftserver overlays

* Patch postgres dc for ImageChange trigger

Co-authored-by: Maulik Shah <maulik.shah1995@gmail.com>
maulikjs added a commit that referenced this pull request Nov 3, 2020
* Add Thriftserver base

* Add dev overlays

* Add stage overlays

* Add prod-* overlays

* Cleanup: remove vars

* Enlist thriftserver overlays

* Patch postgres dc for ImageChange trigger

Co-authored-by: Maulik Shah <maulik.shah1995@gmail.com>
maulikjs added a commit that referenced this pull request Nov 3, 2020
* Add Thriftserver base

* Add dev overlays

* Add stage overlays

* Add prod-* overlays

* Cleanup: remove vars

* Enlist thriftserver overlays

* Patch postgres dc for ImageChange trigger

Co-authored-by: Maulik Shah <maulik.shah1995@gmail.com>
maulikjs added a commit that referenced this pull request Nov 3, 2020
* Add Thriftserver base

* Add dev overlays

* Add stage overlays

* Add prod-* overlays

* Cleanup: remove vars

* Enlist thriftserver overlays

* Patch postgres dc for ImageChange trigger

Co-authored-by: Maulik Shah <maulik.shah1995@gmail.com>
maulikjs added a commit that referenced this pull request Nov 3, 2020
* Add Thriftserver base

* Add dev overlays

* Add stage overlays

* Add prod-* overlays

* Cleanup: remove vars

* Enlist thriftserver overlays

* Patch postgres dc for ImageChange trigger

Co-authored-by: Maulik Shah <maulik.shah1995@gmail.com>
sesheta pushed a commit that referenced this pull request Nov 4, 2020
* Add data-catalog

* Data catalog - HUE (#27)

* Split hue base to manageable yaml files

* Add dev overlays for Hue

* Remove hue base manifest dump

* Add stage overlays

* Add prod environments

* Change namespaces in hue.ini in overlays

* Fix selectors for Services

* Fix s/name/secretName/ in volumes

* Cleanup secret-generator names in overlays

* Fix: wrong indent

* Patch mysql-dc for ImageChange trigger

* Data Catalog - Thriftserver (#29)

* Add Thriftserver base

* Add dev overlays

* Add stage overlays

* Add prod-* overlays

* Cleanup: remove vars

* Enlist thriftserver overlays

* Patch postgres dc for ImageChange trigger

Co-authored-by: Maulik Shah <maulik.shah1995@gmail.com>

* Fix selectors in deploymentconfigs

* Fix selectors in services

* Fix selectors in data catalog again

* Reset hue.ini to include all fields

Co-authored-by: Tomáš Coufal <7453394+tumido@users.noreply.github.com>
Co-authored-by: Tomas Coufal <tcoufal@redhat.com>
@tumido tumido deleted the thriftserver branch November 11, 2020 11:56
chauhankaranraj pushed a commit to chauhankaranraj/idh-manifests that referenced this pull request Jan 19, 2021
* Add data-catalog

* Data catalog - HUE (AICoE#27)

* Split hue base to manageable yaml files

* Add dev overlays for Hue

* Remove hue base manifest dump

* Add stage overlays

* Add prod environments

* Change namespaces in hue.ini in overlays

* Fix selectors for Services

* Fix s/name/secretName/ in volumes

* Cleanup secret-generator names in overlays

* Fix: wrong indent

* Patch mysql-dc for ImageChange trigger

* Data Catalog - Thriftserver (AICoE#29)

* Add Thriftserver base

* Add dev overlays

* Add stage overlays

* Add prod-* overlays

* Cleanup: remove vars

* Enlist thriftserver overlays

* Patch postgres dc for ImageChange trigger

Co-authored-by: Maulik Shah <maulik.shah1995@gmail.com>

* Fix selectors in deploymentconfigs

* Fix selectors in services

* Fix selectors in data catalog again

* Reset hue.ini to include all fields

Co-authored-by: Tomáš Coufal <7453394+tumido@users.noreply.github.com>
Co-authored-by: Tomas Coufal <tcoufal@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants