CI speed-ups #706

dzuelke · 2024-03-08T13:13:40Z

The Hatchet tests do a lot of heroku runs to check concurrency calculation, config handling, and other basic stuff on boot. They are performed for both Apache HTTPD and Nginx, with and without blackfire and newrelic installed, etc etc.

This causes a lot of app creations, and a lot of heroku run calls, which overall, takes time.

These changes collapse many of the expensive runs into a single one, and then split the output again to check the results in individual examples.

Turns out the waitforit.sh test helper had a race condition (that did not affect us before since there weren't multiple "runs inside a run"), which is now fixed, too.

In addition, some splitting of Composer and CI related tests brings the maximum test file time way down, and with parallel_tests grouping, the overall execution time drops significantly.

The move to Hatchet 8 a few days ago already dropped the overall CI execution time from 6-7 minutes down to 5-6 minutes (since no app cleanups after the runs are necessary anymore).

Now we're consistenly in the 3-4 minute range, getting as low as 2:45 minutes.

The test diffs are obviously a bit terrible to read, since indentation changed, but not the test contents themselves. For easier review, I compared the list of tests that get executed, to demonstrate that nothing is accidentally dropped with this restructuring.

There are now ten more tests that get executed; condensed list for heroku-20:

diff --git a/h20 b/h20
index bde3d1ce..8e6eb777 100644
--- a/h20
+++ b/h20
@@ -1,0 +2 @@ A PHP application using ext-blackfire and, as its agent, buildpack blackfireio/i
+A PHP application using ext-blackfire and, as its agent, buildpack blackfireio/integration-heroku explicitly during boot logs info messages about agent startup
@@ -4,0 +6 @@ A PHP application using ext-blackfire and, as its agent, buildpack blackfireio/i
+A PHP application using ext-blackfire and, as its agent, buildpack blackfireio/integration-heroku with default BLACKFIRE_LOG_LEVEL during boot does not log info messages about agent startup
@@ -7,0 +10 @@ A PHP application using ext-blackfire and, as its agent, buildpack blackfireio/i
+A PHP application using ext-blackfire and, as its agent, buildpack blackfireio/integration-heroku implicitly during boot logs info messages about agent startup
@@ -10,0 +14 @@ A PHP application using ext-blackfire and, as its agent, our blackfire package e
+A PHP application using ext-blackfire and, as its agent, our blackfire package explicitly during boot logs info messages about agent startup
@@ -13,0 +18 @@ A PHP application using ext-blackfire and, as its agent, our blackfire package w
+A PHP application using ext-blackfire and, as its agent, our blackfire package without BLACKFIRE_SERVER_TOKEN during boot logs info messages about agent startup
@@ -16,0 +22 @@ A PHP application using ext-blackfire and, as its agent, our blackfire package w
+A PHP application using ext-blackfire and, as its agent, our blackfire package with default BLACKFIRE_LOG_LEVEL during boot does not log info messages about agent startup
@@ -19,0 +26 @@ A PHP application using ext-blackfire and, as its agent, our blackfire package i
+A PHP application using ext-blackfire and, as its agent, our blackfire package implicitly during boot logs info messages about agent startup
@@ -32 +39 @@ A PHP application on Heroku CI installs dev dependencies and caches them
-A PHP application on Heroku CI has zend.assertions enabled
+A PHP application on Heroku CI has zend.assertions enabled and auto-runs a composer.json 'test' script entry
@@ -34 +40,0 @@ A PHP application on Heroku CI fails to auto-run tests if nothing suitable is fo
-A PHP application on Heroku CI specifying a composer.json 'test' script entry executes 'composer test'
@@ -49,0 +56 @@ A PHP application using New Relic explicitly does not start New Relic daemon dur
+A PHP application using New Relic explicitly during boot logs info messages about daemon startup
@@ -53,0 +61 @@ A PHP application using New Relic without NEW_RELIC_LICENSE_KEY does not start N
+A PHP application using New Relic without NEW_RELIC_LICENSE_KEY during boot logs info messages about daemon startup
@@ -57,0 +66 @@ A PHP application using New Relic with default NEW_RELIC_LOG_LEVEL does not star
+A PHP application using New Relic with default NEW_RELIC_LOG_LEVEL during boot does not log info messages about daemon startup
@@ -61,0 +71 @@ A PHP application using New Relic implicitly does not start New Relic daemon dur
+A PHP application using New Relic implicitly during boot logs info messages about daemon startup
@@ -490 +500 @@ A PHP 8.3 application with long-running requests that uses PHP 8.3 and the nginx
-A PHP application like the Heroku Getting Started guide example for PHP deploys and works
+A PHP application like the Heroku Getting Started guide example for PHP deploys, works, and re-uses cached dependencies on stack change

The stack change cache test, now combined with another test, ran on heroku-22 only before, so it shows up for that stack as a deletion:

diff --git a/h22 b/h22
index 68194d67..7a48007c 100644
--- a/h22
+++ b/h22
@@ -1,0 +2 @@ A PHP application using ext-blackfire and, as its agent, buildpack blackfireio/i
+A PHP application using ext-blackfire and, as its agent, buildpack blackfireio/integration-heroku explicitly during boot logs info messages about agent startup
@@ -4,0 +6 @@ A PHP application using ext-blackfire and, as its agent, buildpack blackfireio/i
+A PHP application using ext-blackfire and, as its agent, buildpack blackfireio/integration-heroku with default BLACKFIRE_LOG_LEVEL during boot does not log info messages about agent startup
@@ -7,0 +10 @@ A PHP application using ext-blackfire and, as its agent, buildpack blackfireio/i
+A PHP application using ext-blackfire and, as its agent, buildpack blackfireio/integration-heroku implicitly during boot logs info messages about agent startup
@@ -10,0 +14 @@ A PHP application using ext-blackfire and, as its agent, our blackfire package e
+A PHP application using ext-blackfire and, as its agent, our blackfire package explicitly during boot logs info messages about agent startup
@@ -13,0 +18 @@ A PHP application using ext-blackfire and, as its agent, our blackfire package w
+A PHP application using ext-blackfire and, as its agent, our blackfire package without BLACKFIRE_SERVER_TOKEN during boot logs info messages about agent startup
@@ -16,0 +22 @@ A PHP application using ext-blackfire and, as its agent, our blackfire package w
+A PHP application using ext-blackfire and, as its agent, our blackfire package with default BLACKFIRE_LOG_LEVEL during boot does not log info messages about agent startup
@@ -19,0 +26 @@ A PHP application using ext-blackfire and, as its agent, our blackfire package i
+A PHP application using ext-blackfire and, as its agent, our blackfire package implicitly during boot logs info messages about agent startup
@@ -32 +39 @@ A PHP application on Heroku CI installs dev dependencies and caches them
-A PHP application on Heroku CI has zend.assertions enabled
+A PHP application on Heroku CI has zend.assertions enabled and auto-runs a composer.json 'test' script entry
@@ -34,2 +40,0 @@ A PHP application on Heroku CI fails to auto-run tests if nothing suitable is fo
-A PHP application on Heroku CI specifying a composer.json 'test' script entry executes 'composer test'
-A PHP application that has its stack changed re-uses cached dependencies from the prior stack build
@@ -47,0 +53 @@ A PHP application using New Relic explicitly does not start New Relic daemon dur
+A PHP application using New Relic explicitly during boot logs info messages about daemon startup
@@ -51,0 +58 @@ A PHP application using New Relic without NEW_RELIC_LICENSE_KEY does not start N
+A PHP application using New Relic without NEW_RELIC_LICENSE_KEY during boot logs info messages about daemon startup
@@ -55,0 +63 @@ A PHP application using New Relic with default NEW_RELIC_LOG_LEVEL does not star
+A PHP application using New Relic with default NEW_RELIC_LOG_LEVEL during boot does not log info messages about daemon startup
@@ -59,0 +68 @@ A PHP application using New Relic implicitly does not start New Relic daemon dur
+A PHP application using New Relic implicitly during boot logs info messages about daemon startup
@@ -280 +289 @@ A PHP 8.3 application with long-running requests that uses PHP 8.3 and the nginx
-A PHP application like the Heroku Getting Started guide example for PHP deploys and works
+A PHP application like the Heroku Getting Started guide example for PHP deploys, works, and re-uses cached dependencies on stack change

GUS-W-15212157

Useful e.g. for comparing what got executed with different runs. Needs --first-is-1 for parallel rspec, which also fixes the first group prefix to be [TEST GROUP 1] instead of [TEST GROUP ]

Speeds up tests

…ase fails Helps with debugging in cases where a shutdown in a prior run somehow goes wrong, and web server or PHP-FPM fail to start again (since something is already/still listening on the port).

The purpose of this 'program' is to wait for output from a launched program, but with a timeout to prevent hanging forever. Its key feature is the ability to detect when it is part of a pipeline, and to then write something to that pipeline to signal that the expected output is there - this allows constructs like 'waitforit.sh 10 someprogram | { read; do_something_with_someprogram; }'. The previous solution for the internal pipeline signaling is, however, subject to certain race conditions, and then programs might not be terminated correctly, or the pipe signal loop may hang forever. It's simpler overall to just launch two 'timeout's, for the program and for the output grepping, and wait for either or both to terminate.

Speeds stuff up for composer_spec on heroku-22

For better parallelism

Speeds things up for ci_spec

Those are the slowest tests, so it helps a lot with overall time and with parallelization

It's where it belongs really (gets .gitignore-d there, too).

obsolete since fb28585

dzuelke added 17 commits March 8, 2024 03:05

Print sorted list of executed examples after tests

6d1ce07

Useful e.g. for comparing what got executed with different runs. Needs --first-is-1 for parallel rspec, which also fixes the first group prefix to be [TEST GROUP 1] instead of [TEST GROUP ]

Fewer 'heroku run' calls in php_base_shared

5429b24

Speeds up tests

Fewer 'heroku run' calls in php_default_spec

fb2b1dd

Speeds up tests

Fewer 'heroku run' calls in php_boot_shared

c2aa041

Speeds up tests

Fewer 'heroku run' calls in php_concurrency_shared

6f4cac4

Speeds up tests

Fewer 'heroku run' calls in blackfire_shared

28bffba

Speeds up tests

Fewer 'heroku run' calls in newrelic_spec

72d37ad

Speeds up tests

Print output from same run's previous commands when a php_boot_spec c…

5e1a6a9

…ase fails Helps with debugging in cases where a shutdown in a prior run somehow goes wrong, and web server or PHP-FPM fail to start again (since something is already/still listening on the port).

Speed up platform_spec by bailing out of some builds early

a7bae1f

Move stack change test to php_default_spec

dd23fe8

Speeds stuff up for composer_spec on heroku-22

Split composer_spec into v1 and v2

c387f6e

For better parallelism

Combine 'composer test' and zend.assertions CI tests into one

9740e75

Speeds things up for ci_spec

Split ci_frameworks_spec into two

bd46b5d

Those are the slowest tests, so it helps a lot with overall time and with parallelization

Slightly adjust test runtimes for better balancing

79b416e

Write runtime log to test/var/log/

4ab70d2

It's where it belongs really (gets .gitignore-d there, too).

drop outdated comment

f307602

obsolete since fb28585

dzuelke requested a review from a team as a code owner March 8, 2024 13:13

edmorley approved these changes Mar 8, 2024

View reviewed changes

dzuelke merged commit e74f187 into main Mar 8, 2024
5 checks passed

dzuelke deleted the ci-herokurun-speedups branch March 8, 2024 16:14

heroku-linguist bot mentioned this pull request Mar 28, 2024

Prepare release v248 #708

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI speed-ups #706

CI speed-ups #706

dzuelke commented Mar 8, 2024

CI speed-ups #706

CI speed-ups #706

Conversation

dzuelke commented Mar 8, 2024