Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(streaming): onboard streaming module #1024

Merged
merged 25 commits into from
Jun 8, 2024
Merged

feat(streaming): onboard streaming module #1024

merged 25 commits into from
Jun 8, 2024

Conversation

tyyzqmf
Copy link
Contributor

@tyyzqmf tyyzqmf commented Apr 7, 2024 •
edited
Loading

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

Summary

截屏2024-06-03 17 10 35
  1. If pipeline ingestion server type is KDS，then you can choose to enable streaming.
  2. If the pipeline is an old version, this UI component will not be visible.
  3. The old version pipeline needs to upgrade the console first, then edit the pipeline and enable this streaming module.
截屏2024-04-08 11 25 30

In the application list, users can turn on and off streaming for specified apps, and real-time streaming stack will only process the enabled app data.

Implementation highlights

(describe how the merge request does for feature changes, share the RFC link if it has)

Test checklist

  • add new test cases
  • all code changes are covered by unit tests
  • end-to-end tests
    • deploy web console with CloudFront + S3 + API gateway
    • deploy web console within VPC
    • deploy ingestion server
      • with MSK sink
      • with KDS sink
      • with S3 sink
    • deploy data processing
    • deploy data modeling
      • new Redshift Serverless
      • provisioned Redshift
      • Athena
    • deploy with reporting
    • streaming ingestion
      • with Redshift Serverless
      • with provisioned Redshift

Is it a breaking change

  • add parameters without default value in stack
  • introduce new service permission in stack
  • introduce new top level stack module

Miscellaneous

  • introduce new symbol link source file(s) to be shared among infra code, web console frontend, and web console backend

@tyyzqmf tyyzqmf requested a review from zxkane April 7, 2024 02:14
@tyyzqmf tyyzqmf self-assigned this Apr 7, 2024
@tyyzqmf tyyzqmf marked this pull request as draft April 7, 2024 02:14
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 7, 2024

Streaming ingestion flink application's test coverage

Overall Project 94.57% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 7, 2024

Data Pipeline spark application's test coverage

Overall Project 97.07% 🍏

There is no coverage information present for the Files changed

@tyyzqmf tyyzqmf marked this pull request as ready for review April 7, 2024 03:08
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 7, 2024

Streaming ingestion flink application's test coverage

Overall Project 94.57% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 7, 2024

Data Pipeline spark application's test coverage

Overall Project 97.07% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 7, 2024

Streaming ingestion flink application's test coverage

Overall Project 94.57% 🍏

There is no coverage information present for the Files changed

1 similar comment
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 7, 2024

Streaming ingestion flink application's test coverage

Overall Project 94.57% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 7, 2024

Data Pipeline spark application's test coverage

Overall Project 97.07% 🍏

There is no coverage information present for the Files changed

1 similar comment
@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 7, 2024

Data Pipeline spark application's test coverage

Overall Project 97.07% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 7, 2024

Streaming ingestion flink application's test coverage

Overall Project 94.57% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 7, 2024

Data Pipeline spark application's test coverage

Overall Project 97.07% 🍏

There is no coverage information present for the Files changed

@zxkane zxkane changed the title feat: onboard streaming module feat(streaming): onboard streaming module Apr 8, 2024
@zxkane zxkane requested a review from luorobin-a2z April 8, 2024 02:59
zxkane
zxkane requested changes Apr 8, 2024
View reviewed changes
Copy link
Contributor

@zxkane zxkane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • polish UI wordings

frontend/src/apis/application.ts Show resolved Hide resolved
frontend/src/pages/projects/detail/comp/ProjectPipeline.tsx Outdated Show resolved Hide resolved
frontend/src/ts/utils.ts Show resolved Hide resolved
frontend/src/ts/utils.ts Outdated Show resolved Hide resolved
@zxkane zxkane marked this pull request as draft May 10, 2024 02:22
@github-actions GitHub Actions
Copy link

Etl common library's test coverage

Overall Project 92.62% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

Streaming ingestion flink application's test coverage

Overall Project 94.57% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

Data Pipeline spark application's test coverage

Overall Project 98.17% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

Etl common library's test coverage

Overall Project 92.62% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

Streaming ingestion flink application's test coverage

Overall Project 94.57% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

Data Pipeline spark application's test coverage

Overall Project 98.18% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

Etl common library's test coverage

Overall Project 92.62% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

Streaming ingestion flink application's test coverage

Overall Project 94.57% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

Data Pipeline spark application's test coverage

Overall Project 98.18% 🍏

There is no coverage information present for the Files changed

@zxkane
Copy link
Contributor

zxkane commented May 11, 2024

SonarQube Quality Gate Result

  • Result: ✅ OK
  • Triggered by @tyyzqmf on pull_request
Metric Status Value Error Threshold
Reliability rating ✅ OK 1 > 1
Security rating ✅ OK 1 > 1
Sqale rating ✅ OK 1 > 1
Coverage ✅ OK 84 < 80
Duplicated lines density ✅ OK 5 > 30
Blocker violations ✅ OK 0 > 0
Bugs ✅ OK 0 > 0
Code smells ✅ OK 7 > 40
Critical violations ✅ OK 0 > 0
Major violations ✅ OK 0 > 0
Vulnerabilities ✅ OK 0 > 0

View on SonarQube

updated: 5/11/2024, 10:33:35 (UTC+0)

@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 7, 2024

Etl common library's test coverage

Overall Project 92.59% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 7, 2024

Streaming ingestion flink application's test coverage

Overall Project 94.57% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 7, 2024

Data Pipeline spark application's test coverage

Overall Project 98.12% 🍏

There is no coverage information present for the Files changed

@zxkane
Copy link
Contributor

zxkane commented Jun 7, 2024

SonarQube Quality Gate Result

  • Result: ✅ OK
  • Triggered by @zxkane on pull_request
Metric Status Value Error Threshold
Reliability rating ✅ OK 1 > 1
Security rating ✅ OK 1 > 1
Sqale rating ✅ OK 1 > 1
Coverage ✅ OK 83.70 < 80
Duplicated lines density ✅ OK 5.20 > 30
Blocker violations ✅ OK 0 > 0
Bugs ✅ OK 0 > 0
Code smells ✅ OK 8 > 40
Critical violations ✅ OK 0 > 0
Major violations ✅ OK 0 > 0
Vulnerabilities ✅ OK 0 > 0

View on SonarQube

updated: 6/7/2024, 05:04:50 (UTC+0)

zxkane
zxkane requested changes Jun 7, 2024
View reviewed changes
frontend/public/locales/zh-CN/application.json Outdated Show resolved Hide resolved
frontend/src/ts/utils.ts Outdated Show resolved Hide resolved
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 8, 2024

Etl common library's test coverage

Overall Project 92.59% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 8, 2024

Streaming ingestion flink application's test coverage

Overall Project 94.57% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 8, 2024

Data Pipeline spark application's test coverage

Overall Project 98.12% 🍏

There is no coverage information present for the Files changed

@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 8, 2024

Code Coverage

Package Line Rate Branch Rate Health
src 100% 82% ✔
src.analytics 100% 100% ✔
src.analytics.lambdas 96% 89% ✔
src.analytics.lambdas.clear-expired-events-workflow 100% 100% ✔
src.analytics.lambdas.custom-resource 96% 87% ✔
src.analytics.lambdas.load-data-workflow 99% 90% ✔
src.analytics.lambdas.refresh-materialized-views-workflow 96% 85% ✔
src.analytics.lambdas.scan-metadata-workflow 97% 88% ✔
src.analytics.lambdas.sql-execution-sfn 93% 60% ✔
src.analytics.lambdas.user-segments-workflow 96% 77% ✔
src.analytics.private 100% 92% ✔
src.analytics.private.athena 100% 100% ✔
src.analytics.private.segments 100% 100% ✔
src.base-lib.src 100% 100% ✔
src.base-lib.src.auth 89% 65% ✔
src.base-lib.src.common 97% 72% ✔
src.base-lib.src.constant 100% 100% ✔
src.base-lib.src.model 100% 100% ✔
src.common 98% 80% ✔
src.common.lambda 97% 75% ✔
src.control-plane 99% 71% ✔
src.control-plane.auth 93% 12% ✔
src.control-plane.backend 100% 84% ✔
src.control-plane.backend.lambda.api 97% 33% ✔
src.control-plane.backend.lambda.api.common 90% 78% ✔
src.control-plane.backend.lambda.api.middle-ware 89% 83% ✔
src.control-plane.backend.lambda.api.model 95% 54% ✔
src.control-plane.backend.lambda.api.router 98% 97% ✔
src.control-plane.backend.lambda.api.service 84% 66% ✔
src.control-plane.backend.lambda.api.service.quicksight 93% 81% ✔
src.control-plane.backend.lambda.api.service.segments 95% 76% ✔
src.control-plane.backend.lambda.api.store.aws 86% 55% ✔
src.control-plane.backend.lambda.api.store.dynamodb 90% 78% ✔
src.control-plane.backend.lambda.api.test.api 99% 81% ✔
src.control-plane.backend.lambda.api.test.api.segments 100% 100% ✔
src.control-plane.backend.lambda.batch-insert-ddb 94% 86% ✔
src.control-plane.backend.lambda.listen-stack-status 81% 52% ✔
src.control-plane.backend.lambda.listen-state-status 91% 50% ✔
src.control-plane.backend.lambda.sfn-action 88% 49% ✔
src.control-plane.backend.lambda.sfn-workflow 80% 67% ➖
src.control-plane.backend.layer.lambda-web-adapter 98% 40% ✔
src.control-plane.private 100% 100% ✔
src.data-pipeline 100% 100% ✔
src.data-pipeline.lambda.copy-assets 97% 90% ✔
src.data-pipeline.lambda.emr-job-state-listener 96% 91% ✔
src.data-pipeline.lambda.emr-job-submitter 99% 77% ✔
src.data-pipeline.lambda.emr-serverless-app 96% 86% ✔
src.data-pipeline.lambda.partition-syncer 98% 89% ✔
src.data-pipeline.tables 100% 100% ✔
src.data-pipeline.utils 100% 100% ✔
src.ingestion-server.custom-resource 100% 100% ✔
src.ingestion-server.custom-resource.delete-ecs-cluster 98% 90% ✔
src.ingestion-server.custom-resource.update-alb-rules 100% 78% ✔
src.ingestion-server.kafka-s3-connector 100% 100% ✔
src.ingestion-server.kafka-s3-connector.custom-resource.kafka-s3-sink-connector 96% 82% ✔
src.ingestion-server.kinesis-data-stream 100% 100% ✔
src.ingestion-server.kinesis-data-stream.kinesis-to-s3-lambda 100% 100% ✔
src.ingestion-server.kinesis-data-stream.private 100% 100% ✔
src.ingestion-server.server 98% 89% ✔
src.ingestion-server.server-v2 100% 79% ✔
src.ingestion-server.server-v2.private 100% 82% ✔
src.ingestion-server.server.private 100% 89% ✔
src.metrics 100% 100% ✔
src.metrics.custom-resource.add-sns-subscription 95% 86% ✔
src.metrics.custom-resource.get-interval 99% 97% ✔
src.metrics.custom-resource.put-dashboard 97% 84% ✔
src.metrics.custom-resource.set-metrics-widgets 98% 93% ✔
src.private 96% 73% ✔
src.reporting 100% 80% ✔
src.reporting.lambda.custom-resource.quicksight 94% 81% ✔
src.reporting.private 90% 54% ✔
src.streaming-ingestion 100% 100% ✔
src.streaming-ingestion.lambdas.custom-resource 86% 63% ✔
src.streaming-ingestion.private 99% 89% ✔
src.streaming-ingestion.redshift 100% 100% ✔
test 96% 83% ✔
test.common 93% 100% ✔
test.control-plane 97% 64% ✔
test.ingestion-server.server 97% 88% ✔
Summary 96% (76021 / 79287) 74% (6407 / 8693) ✔

Minimum allowed line rate is 60%

@zxkane
Copy link
Contributor

zxkane commented Jun 8, 2024

SonarQube Quality Gate Result

  • Result: ✅ OK
  • Triggered by @tyyzqmf on pull_request
Metric Status Value Error Threshold
Reliability rating ✅ OK 1 > 1
Security rating ✅ OK 1 > 1
Sqale rating ✅ OK 1 > 1
Coverage ✅ OK 83.50 < 80
Duplicated lines density ✅ OK 5.20 > 30
Blocker violations ✅ OK 0 > 0
Bugs ✅ OK 0 > 0
Code smells ✅ OK 8 > 40
Critical violations ✅ OK 0 > 0
Major violations ✅ OK 0 > 0
Vulnerabilities ✅ OK 0 > 0

View on SonarQube

updated: 6/8/2024, 02:53:53 (UTC+0)

@zxkane zxkane added this pull request to the merge queue Jun 8, 2024
Merged via the queue into main with commit a4294ee Jun 8, 2024
18 checks passed
@zxkane zxkane deleted the realtime-control-plane branch June 8, 2024 15:57
amliuyong pushed a commit that referenced this pull request Jun 11, 2024
@tyyzqmf @zxkane @amliuyong
feat(streaming): onboard streaming module (#1024)
3263a2c 
Signed-off-by: github-actions <github-actions@github.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Meng Xin Zhu <843303+zxkane@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zxkane zxkane zxkane approved these changes

@chenhaiyun chenhaiyun chenhaiyun approved these changes

@luorobin-a2z luorobin-a2z Awaiting requested review from luorobin-a2z

Assignees

@tyyzqmf tyyzqmf

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@tyyzqmf @zxkane @chenhaiyun