Require checking/rescheduling of nightly jobs during releases (elasti…

…c#187)

Also clarify schedule of nightlies and revision picking strategy.

.github/ISSUE_TEMPLATE/release-benchmarks.md

@@ -7,6 +7,12 @@ assignees: ''
 
 ---
 
+IMPORTANT NOTE
+-------------------
+For every task below, after scheduling the job, remember to cancel nightly jobs on [group-1](https://elasticsearch-ci.elastic.co/view/All/job/elastic+elasticsearch+master+macrobenchmark-periodic-group-1/) and [group-2](https://elasticsearch-ci.elastic.co/view/All/job/elastic+elasticsearch+master+macrobenchmark-periodic-group-2/) that get autoscheduled at 20:00 UTC.
+If your released job seems to be still running, cancel them (use the x button after logging in on Jenkins) and reschedule using the correct effective start date.
+More details in [night-rally docs](https://github.com/elastic/night-rally/blob/master/42.md#what-time-do-the-nightly-benchmarks-start-what-elasticsearch-commit-do-they-choose).
+
 - [ ] mode: `release`, release_license: `oss` - triggered by `your user handle here`  as `link to all triggered builds here`
 - [ ] mode: `release`, release_license: `basic` - triggered by `your user handle here`  as `link to all triggered builds here`
 - [ ] mode: `release`, RELEASE_LICENSE: `trial` + RELEASE_X_PACK_COMPONENTS: `security` - triggered by `your user handle here`  as `link to all triggered builds here`

42.md

@@ -12,6 +12,14 @@ The two former utilize two groups of machines in parallel, due to the total time
 
 Our long-running benchmarks also run on bare-metal machines but are just triggered directly on the machines as it does not make too much sense to leave a Jenkins build running for one month. Results from long-running benchmarks can be found in this [Kibana dashboard](https://ela.st/elasticsearch-long-running-benchmarks).
 
+## What time do the nightly benchmarks start? What Elasticsearch commit do they choose?
+
+This is defined in the corresponding Jenkins Jobs: [group-1](https://github.com/elastic/infra/blob/ae12944b83c66a54f4c43d76c785aed8a638f10e/ci/jjb/elasticsearch-ci/globals/templates/elastic-elasticsearch/macrobenchmarks-group-1.yml#L70) and [group-2](https://github.com/elastic/infra/blob/ae12944b83c66a54f4c43d76c785aed8a638f10e/ci/jjb/elasticsearch-ci/globals/templates/elastic-elasticsearch/macrobenchmarks-group-2.yml#L70).
+
+These jobs don't set an effective start date, so night-rally will pick the [current date time as effective start date](https://github.com/elastic/night-rally/blob/a71c4fe3d61015b2bf8654ff88a59efd83052bca/night_rally.sh#L45).
+
+The Elasticsearch revision will be the closest commit to the effective start date (see [--revision in Rally](https://esrally.readthedocs.io/en/stable/command_line_reference.html#revision)). 
+
 ### How are the benchmarks run? Can I also execute them?
 
 All macrobenchmarks are run with [Rally](https://github.com/elastic/rally) which is open source. Rally also has a [user documentation](http://esrally.readthedocs.io/en/latest/).