Add buildkite annotation when notebook parallelism setting is wrong

[huonw]

If the number of notebooks doesn't match the parallelism setting used for testing notebooks, all of the steps will fail, and one needs to go into them to deduce what the problem was, and then interpret the message. This PR (a) rewrites the message to be a bit clearer, including mentioning the common case of needing to merge with develop, and (b) adds it as an annotation at the top of the build for convenience.

Example

https://buildkite.com/stellar/stellargraph-public/builds/1628

[codeclimate]

Code Climate has analyzed commit 7067f61 and detected 0 issues on this pull request.

View more on Code Climate.

[stellar-graph-bot]

Codecov Report

Merging #916 into develop will not change coverage by %.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           develop    #916   +/-   ##
=======================================
  Coverage     84.4%   84.4%           
=======================================
  Files           52      52           
  Lines         5080    5080           
=======================================
  Hits          4288    4288           
  Misses         792     792           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 66de381...7067f61. Read the comment docs.

[codecov-io]

Codecov Report

Merging #916 into develop will not change coverage.
The diff coverage is 44.8%.

@@           Coverage Diff           @@
##           develop    #916   +/-   ##
=======================================
  Coverage     84.4%   84.4%           
=======================================
  Files           52      52           
  Lines         5080    5080           
=======================================
  Hits          4288    4288           
  Misses         792     792

Impacted Files	Coverage Δ
stellargraph/core/utils.py	94.3% <100%> (ø)	⬆️
stellargraph/core/graph.py	98.6% <100%> (ø)	⬆️
stellargraph/core/convert.py	100% <100%> (ø)	⬆️
stellargraph/mapper/graphwave_generator.py	25.6% <25.6%> (ø)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fa2e8c4...7067f61. Read the comment docs.

[kieranricardo]

just to clarify, when will merging with develop fix this?

[huonw]

When a PR adding or removing notebooks was merged into develop, and the current build's commit doesn't inherit from that merged commit, e.g.: if the git DAG for develop and the PR in question looks like

D (develop)
|
|  P (PR)
| /
B
|
older
|
commits
|
...

Suppose commit D on develop adds a notebook and so has parallelism: 40, vs. the PR commit P that has parallelism: 39. When P is built against D, the pipeline.yml will be uploaded from P (and so parallelism: 39). The notebook steps will be run after doing a transient merge with D on the CI machines, the plugins anchor pulls in the https://github.com/stellargraph/github-merged-pr-buildkite-plugin plugin:

stellargraph/.buildkite/pipeline.yml

Lines 82 to 89 in 913cf97

	- label: ":python::book: test notebooks %n"
	<<: *timeout
	key: "test-notebooks"
	depends_on: "runner-3_6"
	parallelism: 36
	command: ".buildkite/steps/test-demo-notebooks.sh"
	plugins:
	<<: *plugins

After doing the merge with D, the code being tested for P will have 40 notebooks (the merge will bring in that extra notebook), and thus the parallelism setting of 39 will be wrong. If an explicit merge is done and pushed to the PR, the parallelism in the pipeline.yml will be correct (40):

   M (PR)
 / |
D (develop)
|  | 
|  P
| /
B
|
|
older
|
commits
|
...

[kieranricardo]

ah makes sense, I wasn't aware of the transient merge going on that would pull in extra notebooks from develop 👍

[kieranricardo]

looks good!