Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BEAM-12918] TPC-DS: add Jenkins jobs #15679

Merged
merged 1 commit into from May 10, 2022
Merged

[BEAM-12918] TPC-DS: add Jenkins jobs #15679

merged 1 commit into from May 10, 2022

Conversation

aromanenko-dev
Copy link
Contributor

@aromanenko-dev aromanenko-dev commented Oct 7, 2021 •
edited

Add a Jenkins job to run TPC-DS benchmark against Spark runner.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

ValidatesRunner compliance status (on master branch)

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- Build Status Build Status Build Status Build Status ---
Java Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status 		Build Status
Build Status 		Build Status
Build Status
Build Status 		Build Status
Python --- Build Status
Build Status
Build Status 		Build Status
Build Status 		Build Status Build Status ---
XLang Build Status Build Status Build Status Build Status Build Status ---

Examples testing status on various runners

Lang ULR Dataflow Flink Samza Spark Twister2
Go --- --- --- --- --- --- ---
Java --- Build Status
Build Status
Build Status
--- --- --- --- ---
Python --- --- --- --- --- --- ---
XLang --- --- --- --- --- --- ---

Post-Commit SDK/Transform Integration Tests Status (on master branch)

Go Java Python
Build Status Build Status Build Status
Build Status
Build Status

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status
Build Status
Build Status
Build Status
Build Status 		Build Status Build Status Build Status Build Status
Portable --- Build Status Build Status --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@aromanenko-dev aromanenko-dev force-pushed the BEAM-12918-tpcds-jenkins branch 2 times, most recently from 4f7b551 to 97ff2c3 Compare October 11, 2021 11:37
@aromanenko-dev
Copy link
Contributor Author

Run Seed Job

@aromanenko-dev
Copy link
Contributor Author

Run Spark Runner Tpcds Tests

@aromanenko-dev
Copy link
Contributor Author

Run Spark Runner Tpcds Tests

@aromanenko-dev
Copy link
Contributor Author

Run Spark Runner Tpcds Tests

@aromanenko-dev
Copy link
Contributor Author

Run Seed Job

@aromanenko-dev
Copy link
Contributor Author

Run Spark Runner Tpcds Tests

@aromanenko-dev
Copy link
Contributor Author

Run Seed Job

@aromanenko-dev
Copy link
Contributor Author

Run Spark Runner Tpcds Tests

@aromanenko-dev
Copy link
Contributor Author

Run Seed Job

@aromanenko-dev
Copy link
Contributor Author

Run Spark Runner Tpcds Tests

@aromanenko-dev
Copy link
Contributor Author

Run Seed Job

@aromanenko-dev
Copy link
Contributor Author

Run Spark Runner Tpcds Tests

@aromanenko-dev
Copy link
Contributor Author

Run Spark Runner Nexmark Tests

@aromanenko-dev aromanenko-dev changed the title [WIP][BEAM-12918] TPC-DS: add Jenkins jobs [BEAM-12918] TPC-DS: add Jenkins jobs Feb 2, 2022
@aromanenko-dev
Copy link
Contributor Author

R: @echauchot ptal

echauchot
echauchot requested changes Feb 3, 2022
View reviewed changes
Copy link
Contributor

@echauchot echauchot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Alexey for your work !
I would also add the other Spark runner (structured streaming) which is supported
Can you also add Flink runner ?

.test-infra/jenkins/job_PostCommit_Java_Tpcds_Spark.groovy
import CommonJobProperties as commonJobProperties
import InfluxDBCredentialsHelper

PhraseTriggeringPostCommitBuilder.postCommitJob('beam_PostCommit_Java_Tpcds_Spark',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please also add a phrase for the Spark Structured Streaming Runner

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ping ?

@aromanenko-dev
Copy link
Contributor Author

Run Spark ValidatesRunner

@aaltay
Copy link
Member

aaltay commented Mar 10, 2022

@aromanenko-dev - sorry for the ping. Is this PR still active?

@aromanenko-dev
Copy link
Contributor Author

@aaltay Yep, sorry for delay, I'm going to update it soon.

@aromanenko-dev
Copy link
Contributor Author

Run Seed Job

@aromanenko-dev
Copy link
Contributor Author

Run Spark Runner Tpcds Tests

@aromanenko-dev
Copy link
Contributor Author

Run Seed Job

@aromanenko-dev
Copy link
Contributor Author

Run Spark Runner Tpcds Tests

1 similar comment
@aromanenko-dev
Copy link
Contributor Author

Run Spark Runner Tpcds Tests

@aromanenko-dev
Copy link
Contributor Author

@echauchot Could you take a look one more time? I added different Spark runners and versions to this job.

echauchot
echauchot reviewed May 10, 2022
View reviewed changes
Copy link
Contributor

@echauchot echauchot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the updates @aromanenko-dev. I have 2 questions comparing to Nexmark

.test-infra/jenkins/job_PostCommit_Java_Tpcds_Spark.groovy Show resolved Hide resolved
.test-infra/jenkins/job_PostCommit_Java_Tpcds_Spark.groovy Show resolved Hide resolved
@aromanenko-dev
Copy link
Contributor Author

@echauchot Thanks, answered

echauchot
echauchot approved these changes May 10, 2022
View reviewed changes
Copy link
Contributor

@echauchot echauchot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks.
Merginig

@echauchot echauchot merged commit 5c21fbc into apache:master May 10, 2022
@aromanenko-dev aromanenko-dev deleted the BEAM-12918-tpcds-jenkins branch May 10, 2022 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@echauchot echauchot echauchot approved these changes

Assignees
No one assigned
Labels
infra java
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@aromanenko-dev @aaltay @echauchot