Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve airbyte-metrics support in the Helm chart #16166

Merged
merged 4 commits into from
Sep 2, 2022
Merged

Improve airbyte-metrics support in the Helm chart #16166

merged 4 commits into from
Sep 2, 2022

Conversation

milesarmstrong
Copy link
Contributor

What

This change enables (optional) deployment of the airbyte metrics-reporter via a new Helm chart. It also allows configuration of metrics-related environment variables from the main airbyte Helm chart.

Resolving: #16129

How

  1. Create a new Helm chart which deploys the airbyte metrics-reporter
  2. Update the main airbyte Helm chart to include it as a (disabled by default) dependency.
  3. Updated the main airbyte Helm chart to allow setting of METRIC_CLIENT and OTEL_COLLECTOR_ENDPOINT in the airbyte-env ConfigMap.

Recommended reading order

  • New Helm chart in charts/airbyte-metrics
  • New dependency in (charts/airbyte/Chart.yaml) and values (charts/airbyte/values.yaml) in airbyte Helm chart.
  • New values under global.metrics in (charts/airbyte/values.yaml)

@CLAassistant
Copy link

CLAassistant commented Aug 31, 2022 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions bot added area/platform issues related to the platform kubernetes labels Aug 31, 2022
@milesarmstrong milesarmstrong force-pushed the miles/16129-airbyte-metrics-helm branch from 2f96e26 to 7854d02 Compare August 31, 2022 12:02
xpuska513
xpuska513 approved these changes Sep 2, 2022
View reviewed changes
Copy link
Contributor

@xpuska513 xpuska513 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xpuska513 xpuska513 merged commit a43c098 into airbytehq:master Sep 2, 2022
xpuska513 added a commit that referenced this pull request Sep 2, 2022
@xpuska513
fix: Include #16166
8ae76c6
@milesarmstrong milesarmstrong mentioned this pull request Sep 2, 2022
Merged
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request Sep 2, 2022
Merged
letiescanciano added a commit that referenced this pull request Sep 5, 2022
@letiescanciano
Merge branch 'master' into leti/fix-page-tracking-in-segment
f4afffe 
* master: (47 commits)
  Add email to identify users analytics call (#16327)
  🎉 Source Amazon Ads: improve `config.start_date` validation (#16191)
  Add comments about intermediate state emission (#16262)
  MySQL Source : Standardize spec.json for DB connectors that support log-based CDC replication (#16216)
  MSSQL Source : Standardize spec.json for DB connectors that support log-based CDC replication (#16215)
  Hide a bunch more destination with potential unsecure API access (#16320)
  Skip unit tests when run-tests is false (#16267)
  Hide Destination connections from UI (#16310)
  Add scheduled task to clean up old files from workspace (#16247)
  Source Google Analytics v4: Re-name google analytics connector (#16306)
  🐛 Source Facebook Marketing: remove "end_date" from config if empty value (re-implement #16096) (#16222)
  Fix github action syntax (#16277)
  Re-name google analytics cionnectors (#16287)
  Bump Airbyte version from 0.40.3 to 0.40.4 (#16275)
  Hide ES and Redis destination connectors from Cloud (#16276)
  15700  add tests for PokeAPI (#15701)
  Add ProtocolVersion to StandardDefs (#16237)
  🪟 🔧 🧹 Migrate attempt `bytesSynced` to `totalStats.bytesEmitted` and cleanup `AttemptDetails` component (#16126)
  Improve behavior of password input field (#16011)
  Improve airbyte-metrics support in the Helm chart (#16166)
  ...
xpuska513 added a commit that referenced this pull request Sep 9, 2022
@xpuska513 @davinchia
fix: Update oss charts. Add acceptance tests for helm (#16131)
3b807c8 
* fix: revert extraEnv. Move updated extraEnv to env_vars

* fix: Add node pool selector feature for gke deployments

* fix: hotfix. Fix webapp values and imagepullsecrets

* fix: remove nodeSelector duplicate

* fix: revert imagePullSecrets changes

* fix nit

* fix: fix bootloader imagepullsecrets

* fix: fix image pull secrets again...

* feat: add acceptance tests for helm

* add step to install unzip

* fix incorrect command spelling(helm)

* update slack notification sending for Airbyte CI workflow

* Rename Helm E2E test report name

* fix: Update helm charts

* fix: fix temporal DB Port bug

* fix nits

* do a small testing for sake of science

* Update test chart.yaml with note block

* Revert changes for AC tests

* fix: Include #16166

* fix: add metrics to the ci workflow

* fix: revert changes for temporal service. move them to server service;

* fix: revert fullstory deletion

* fix: change way of defining image tag and make it to be set in global values. Add values.yaml.test for integration testing and local deployment

* fix: udpate docs, update values.yaml

* fix: fix helpers.tpl in all subcharts

* Update chart docs

Co-authored-by: Davin Chia <davinchia@gmail.com>
xpuska513 added a commit that referenced this pull request Sep 9, 2022
@xpuska513 @davinchia
fix: Update helm-docs (#16531)
610628a 
* fix: revert extraEnv. Move updated extraEnv to env_vars

* fix: Add node pool selector feature for gke deployments

* fix: hotfix. Fix webapp values and imagepullsecrets

* fix: remove nodeSelector duplicate

* fix: revert imagePullSecrets changes

* fix nit

* fix: fix bootloader imagepullsecrets

* fix: fix image pull secrets again...

* feat: add acceptance tests for helm

* fix: add step to install unzip

* fix incorrect command spelling(helm)

* fix: update slack notification sending for Airbyte CI workflow

* fix: Rename Helm E2E test report name

* fix: Update helm charts

* fix: fix temporal DB Port bug

* fix nits

* fix: do a small testing for sake of science

* fix: Update test chart.yaml with note block

* fix: Revert changes for AC tests

* fix: Include #16166

* fix: add metrics to the ci workflow

* fix: revert changes for temporal service. move them to server service;

* fix: revert fullstory deletion

* fix: change way of defining image tag and make it to be set in global values. Add values.yaml.test for integration testing and local deployment

* fix: udpate docs, update values.yaml

* fix: fix helpers.tpl in all subcharts

* Update chart docs

* fix: update helm-docs

* fix: revert changes to ingress

Co-authored-by: Davin Chia <davinchia@gmail.com>
robbinhan pushed a commit to robbinhan/airbyte that referenced this pull request Sep 29, 2022
@milesarmstrong @robbinhan
Improve airbyte-metrics support in the Helm chart (airbytehq#16166)
298a96d 
* Add a new airbyte-metrics Helm chart

* Include metrics Chart as a dependency of the main airbyte Chart

* Allow setting METRIC_CLIENT and OTEL_COLLECTOR_ENDPOINT from Helm values

* Actually make metrics sub-chart disabled by default
robbinhan pushed a commit to robbinhan/airbyte that referenced this pull request Sep 29, 2022
@xpuska513 @davinchia @robbinhan
fix: Update oss charts. Add acceptance tests for helm (airbytehq#16131)
ff79bcd 
* fix: revert extraEnv. Move updated extraEnv to env_vars

* fix: Add node pool selector feature for gke deployments

* fix: hotfix. Fix webapp values and imagepullsecrets

* fix: remove nodeSelector duplicate

* fix: revert imagePullSecrets changes

* fix nit

* fix: fix bootloader imagepullsecrets

* fix: fix image pull secrets again...

* feat: add acceptance tests for helm

* add step to install unzip

* fix incorrect command spelling(helm)

* update slack notification sending for Airbyte CI workflow

* Rename Helm E2E test report name

* fix: Update helm charts

* fix: fix temporal DB Port bug

* fix nits

* do a small testing for sake of science

* Update test chart.yaml with note block

* Revert changes for AC tests

* fix: Include airbytehq#16166

* fix: add metrics to the ci workflow

* fix: revert changes for temporal service. move them to server service;

* fix: revert fullstory deletion

* fix: change way of defining image tag and make it to be set in global values. Add values.yaml.test for integration testing and local deployment

* fix: udpate docs, update values.yaml

* fix: fix helpers.tpl in all subcharts

* Update chart docs

Co-authored-by: Davin Chia <davinchia@gmail.com>
robbinhan pushed a commit to robbinhan/airbyte that referenced this pull request Sep 29, 2022
@xpuska513 @davinchia @robbinhan
fix: Update helm-docs (airbytehq#16531)
84804b5 
* fix: revert extraEnv. Move updated extraEnv to env_vars

* fix: Add node pool selector feature for gke deployments

* fix: hotfix. Fix webapp values and imagepullsecrets

* fix: remove nodeSelector duplicate

* fix: revert imagePullSecrets changes

* fix nit

* fix: fix bootloader imagepullsecrets

* fix: fix image pull secrets again...

* feat: add acceptance tests for helm

* fix: add step to install unzip

* fix incorrect command spelling(helm)

* fix: update slack notification sending for Airbyte CI workflow

* fix: Rename Helm E2E test report name

* fix: Update helm charts

* fix: fix temporal DB Port bug

* fix nits

* fix: do a small testing for sake of science

* fix: Update test chart.yaml with note block

* fix: Revert changes for AC tests

* fix: Include airbytehq#16166

* fix: add metrics to the ci workflow

* fix: revert changes for temporal service. move them to server service;

* fix: revert fullstory deletion

* fix: change way of defining image tag and make it to be set in global values. Add values.yaml.test for integration testing and local deployment

* fix: udpate docs, update values.yaml

* fix: fix helpers.tpl in all subcharts

* Update chart docs

* fix: update helm-docs

* fix: revert changes to ingress

Co-authored-by: Davin Chia <davinchia@gmail.com>
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
@milesarmstrong @jhammarstedt
Improve airbyte-metrics support in the Helm chart (airbytehq#16166)
5dae4eb 
* Add a new airbyte-metrics Helm chart

* Include metrics Chart as a dependency of the main airbyte Chart

* Allow setting METRIC_CLIENT and OTEL_COLLECTOR_ENDPOINT from Helm values

* Actually make metrics sub-chart disabled by default
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
@xpuska513 @davinchia @jhammarstedt
fix: Update oss charts. Add acceptance tests for helm (airbytehq#16131)
62fb7da 
* fix: revert extraEnv. Move updated extraEnv to env_vars

* fix: Add node pool selector feature for gke deployments

* fix: hotfix. Fix webapp values and imagepullsecrets

* fix: remove nodeSelector duplicate

* fix: revert imagePullSecrets changes

* fix nit

* fix: fix bootloader imagepullsecrets

* fix: fix image pull secrets again...

* feat: add acceptance tests for helm

* add step to install unzip

* fix incorrect command spelling(helm)

* update slack notification sending for Airbyte CI workflow

* Rename Helm E2E test report name

* fix: Update helm charts

* fix: fix temporal DB Port bug

* fix nits

* do a small testing for sake of science

* Update test chart.yaml with note block

* Revert changes for AC tests

* fix: Include airbytehq#16166

* fix: add metrics to the ci workflow

* fix: revert changes for temporal service. move them to server service;

* fix: revert fullstory deletion

* fix: change way of defining image tag and make it to be set in global values. Add values.yaml.test for integration testing and local deployment

* fix: udpate docs, update values.yaml

* fix: fix helpers.tpl in all subcharts

* Update chart docs

Co-authored-by: Davin Chia <davinchia@gmail.com>
jhammarstedt pushed a commit to jhammarstedt/airbyte that referenced this pull request Oct 31, 2022
@xpuska513 @davinchia @jhammarstedt
fix: Update helm-docs (airbytehq#16531)
3039636 
* fix: revert extraEnv. Move updated extraEnv to env_vars

* fix: Add node pool selector feature for gke deployments

* fix: hotfix. Fix webapp values and imagepullsecrets

* fix: remove nodeSelector duplicate

* fix: revert imagePullSecrets changes

* fix nit

* fix: fix bootloader imagepullsecrets

* fix: fix image pull secrets again...

* feat: add acceptance tests for helm

* fix: add step to install unzip

* fix incorrect command spelling(helm)

* fix: update slack notification sending for Airbyte CI workflow

* fix: Rename Helm E2E test report name

* fix: Update helm charts

* fix: fix temporal DB Port bug

* fix nits

* fix: do a small testing for sake of science

* fix: Update test chart.yaml with note block

* fix: Revert changes for AC tests

* fix: Include airbytehq#16166

* fix: add metrics to the ci workflow

* fix: revert changes for temporal service. move them to server service;

* fix: revert fullstory deletion

* fix: change way of defining image tag and make it to be set in global values. Add values.yaml.test for integration testing and local deployment

* fix: udpate docs, update values.yaml

* fix: fix helpers.tpl in all subcharts

* Update chart docs

* fix: update helm-docs

* fix: revert changes to ingress

Co-authored-by: Davin Chia <davinchia@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xpuska513 xpuska513 xpuska513 approved these changes

Assignees
No one assigned
Labels
area/platform issues related to the platform community kubernetes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@milesarmstrong @CLAassistant @xpuska513 @octavia-squidington-iii