Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding test data sources designed for benchmarking purposes #2382

Merged
merged 28 commits into from Aug 4, 2020
Merged

Adding test data sources designed for benchmarking purposes #2382

merged 28 commits into from Aug 4, 2020

Conversation

guenter-hesse
Copy link
Contributor

@guenter-hesse guenter-hesse commented Jun 30, 2020 •
edited by mtopolnik

Adding this streaming source:

StreamSource<Long> longStream(long eventsPerSecond, long initialDelayMillis)

It is a high-throughput source with precisely controlled timing, to be used for benchmarking Jet's throughput and latency.

Fixes #2244

Checklist

  • Tags Set
  • Milestone Set
  • New public APIs have @Nonnull/@Nullable annotations
  • New public APIs have @since tags in Javadoc

@devOpsHazelcast
Copy link
Contributor

devOpsHazelcast commented Jun 30, 2020 •
edited

CLA assistant check
All committers have signed the CLA.

@devOpsHazelcast
Copy link
Contributor

Can one of the admins verify this patch?

1 similar comment
@devOpsHazelcast
Copy link
Contributor

Can one of the admins verify this patch?

@cangencer
Copy link
Contributor

Thanks @guenter-hesse for the PR! I'm on holiday and will have a look at this when I'm back.

@cangencer
Copy link
Contributor

cangencer commented Jul 15, 2020 •
edited

@guenter-hesse @mtopolnik has improved the benchmark sources since this PR was opened, I think we should replace these with that. What do you think?

@guenter-hesse
Copy link
Contributor Author

@cangencer thank you for the information. I just had a conversation with @mtopolnik on Slack in which we briefly discussed this improved concept. I will update the PR within the next few days correspondingly.

@mtopolnik mtopolnik added this to the 4.3 milestone Jul 21, 2020
@devOpsHazelcast
Copy link
Contributor

Can one of the admins verify this patch?

3 similar comments
@devOpsHazelcast
Copy link
Contributor

Can one of the admins verify this patch?

@devOpsHazelcast
Copy link
Contributor

Can one of the admins verify this patch?

@devOpsHazelcast
Copy link
Contributor

Can one of the admins verify this patch?

cangencer
cangencer approved these changes Jul 23, 2020
View reviewed changes
Copy link
Contributor

@cangencer cangencer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@guenter-hesse looks good to me now, some very minor comments! Can merge after those are fixed once @mtopolnik reviews as well. Thanks for your contribution.

hazelcast-jet-core/src/test/java/com/hazelcast/jet/pipeline/test/TestSourcesTest.java Outdated Show resolved Hide resolved
hazelcast-jet-core/src/main/java/com/hazelcast/jet/pipeline/test/TestSources.java Outdated Show resolved Hide resolved
hazelcast-jet-core/src/test/java/com/hazelcast/jet/pipeline/test/TestSourcesTest.java Outdated Show resolved Hide resolved
@cangencer
Copy link
Contributor

verify

@mtopolnik
Copy link
Contributor

verify

@ghost
Copy link

ghost commented Jul 30, 2020

The job jet-oss-pr-builder of your PR failed (log, artifacts).
Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log file 
--------------------------
---------SUMMARY----------
--------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-checkstyle-plugin:3.1.0:checkstyle (default) on project hazelcast-jet-core: An error has occurred in Checkstyle report generation. Failed during checkstyle execution: There is 1 error reported by Checkstyle 8.29 with /home/jenkins/jenkins_slave/workspace/jet-oss-pr-builder/checkstyle/checkstyle.xml ruleset. -> [Help 1]
--------------------------
---------ERRORS-----------
--------------------------
[ERROR] /home/jenkins/jenkins_slave/workspace/jet-oss-pr-builder/hazelcast-jet-core/src/main/java/com/hazelcast/jet/pipeline/test/LongStreamSourceP.java:25:8: Unused import - com.hazelcast.logging.Logger. [UnusedImports]
--------------------------

@mtopolnik
Copy link
Contributor

verify

@mtopolnik
Copy link
Contributor

verify

@cangencer
Copy link
Contributor

verify

@viliam-durina viliam-durina merged commit 981182e into hazelcast:master Aug 4, 2020
@devOpsHazelcast
Copy link
Contributor

Can one of the admins verify this patch?

5 similar comments
@devOpsHazelcast
Copy link
Contributor

Can one of the admins verify this patch?

@devOpsHazelcast
Copy link
Contributor

Can one of the admins verify this patch?

@devOpsHazelcast
Copy link
Contributor

Can one of the admins verify this patch?

@devOpsHazelcast
Copy link
Contributor

Can one of the admins verify this patch?

@devOpsHazelcast
Copy link
Contributor

Can one of the admins verify this patch?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mtopolnik mtopolnik mtopolnik approved these changes

@cangencer cangencer cangencer approved these changes

@viliam-durina viliam-durina viliam-durina approved these changes

Assignees
No one assigned
Labels
community enhancement
Projects
None yet
Milestone
4.3
Development

Successfully merging this pull request may close these issues.

Create sources for benchmarking
5 participants
@guenter-hesse @devOpsHazelcast @cangencer @mtopolnik @viliam-durina