Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync output of parallel make tasks in CI #1503

Merged
merged 2 commits into from Jan 17, 2020
Merged

Sync output of parallel make tasks in CI #1503

merged 2 commits into from Jan 17, 2020

Conversation

@smlx
Copy link
Contributor

smlx commented Dec 18, 2019

Checklist

  • Affected Issues have been mentioned in the Closing issues section
  • Documentation has been written/updated.
  • Changelog entry has been written

The parallel tasks in CI (-jN) all write to STDOUT at once and the output is garbled. This change syncs the output to avoid the interspersed messages from parallel make tasks.

Changelog Entry

Improvement - Sync output of parallel CI tasks

Closing issues

n/a

@smlx smlx force-pushed the sync-ci-output branch from 4338b52 to f6f564a Jan 3, 2020
Copy link
Contributor

tobybellwood left a comment

much easier - nice!

@tobybellwood tobybellwood added this to the v1.3.0 milestone Jan 5, 2020
@smlx smlx force-pushed the sync-ci-output branch from f6f564a to d89eef4 Jan 8, 2020
@smlx smlx mentioned this pull request Jan 13, 2020
3 of 3 tasks complete
@Schnitzel

This comment has been minimized.

Copy link
Member

Schnitzel commented Jan 13, 2020

This is definitely much easier to read, but unfortunately this biffers the logs while they are happening and only paste them after they are done.
And many times to debug I need realtime logs to see check which test is maybe failing to login into the worker node and check stuff on the CLI.
If make is buffering the logs this won't be possible anymore :(

Is there maybe a possibility that we run multiple sh parallel steps of Jenkins which each runs just a single make test? I guess the question is then how do we explain Jenkins it should only run two tests parallel and not all of them? But I'm sure @smlx you will find a clever way :)

Copy link
Member

Schnitzel left a comment

see comment above

@smlx

This comment has been minimized.

Copy link
Contributor Author

smlx commented Jan 14, 2020

hmm. okay so I've done a bit of research. make's sync-output functionality uses tmpfile(), which uses open(..., O_TMPFILE). This means that the output buffer never actually gets a pathname so we can't tail it.

I think that re-implementing make's parallel functionality in the Jenkinsfile will get messy..

maybe the best way to do this is to have a variable in the Makefile that can be set to control sync-output functionality. By default output can be synced, and you can flip the variable to disable syncing when debugging. What do you think of that solution @Schnitzel?

@Schnitzel

This comment has been minimized.

Copy link
Member

Schnitzel commented Jan 14, 2020

ugh, yea looks like Jenkins doesn't really have a way to do this, see also https://issues.jenkins-ci.org/browse/JENKINS-44085
I guess your suggestion @smlx with a variable would be the easiest, can you add this to this PR?

Avoid the interspersed output from parallel make tasks.
@smlx smlx force-pushed the sync-ci-output branch from 18a13e2 to e30eca4 Jan 17, 2020
@smlx smlx force-pushed the sync-ci-output branch from 4f5ec61 to a58a68a Jan 17, 2020
@smlx

This comment has been minimized.

Copy link
Contributor Author

smlx commented Jan 17, 2020

ready for review again @Schnitzel

@smlx smlx requested a review from Schnitzel Jan 17, 2020
@Schnitzel Schnitzel merged commit 417b71f into master Jan 17, 2020
1 check passed
1 check passed
continuous-integration/jenkins/pr-merge This commit looks good
Details
@smlx smlx deleted the sync-ci-output branch Jan 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.