More github workflow improvements #2506

rwest · 2023-07-24T17:49:19Z

Motivation or Problem

More improvements to the new github workflows

Description of Changes

the regression summary is prepared and added to the workflow job summary for everything but the reference job (the push to main and nightly build).
the annotation workflow is only run for pull requests, not for pushes and scheduled runs.
if the annotation job fails for some reason, it reports the event that triggered it, for debugging..

Testing

I ran things on my main branch. I think they're working - but this workflow stuff has been remarkably hard to get right first time.

The continue-on-error:true makes it think that step succeeded (though it didn't) so the "do on failure" doesn't run, and then the next step fails. Now we let it fail, and at least report in json the information that may let us try something else. One idea is to cange the javascript around line 32 that gets the matchArtifact and if there isn't one then return there and behave differently.

Hoping this works. Also fix a problem with escaping ' inside JSON when logging the github.event.workflow_run context.

Rather than run a pointless "the workflow failed" job, we just skip it.

The Continuous Integration workflow runs on - pushes - pull requests - a schedule - workflow calls from database testing With this change, we create the regression test summary and cat it into the workflow summary for all job types, though it is still only uploaded as an artifact for pull requests.

JacksonBurns

LGTM, though as you said this has been vexing - let's hope this is the fix that gets it.

github-actions · 2023-07-24T21:02:05Z

Regression Testing Results

⚠️ One or more regression tests failed.
Please download the failed results and run the tests locally or check the log to see why.

Detailed regression test results.

Regression test aromatics:

Reference: Execution time (DD:HH:MM:SS): 00:00:01:23
Current: Execution time (DD:HH:MM:SS): 00:00:01:40
Reference: Memory used: 2097.70 MB
Current: Memory used: 2092.66 MB

aromatics Passed Core Comparison ✅

Original model has 15 species.
Test model has 15 species. ✅
Original model has 11 reactions.
Test model has 11 reactions. ✅

aromatics Passed Edge Comparison ✅

Original model has 106 species.
Test model has 106 species. ✅
Original model has 358 reactions.
Test model has 358 reactions. ✅

Observables Test Case: Aromatics Comparison

✅ All Observables varied by less than 0.500 on average between old model and new model in all conditions!

aromatics Passed Observable Testing ✅

Regression test liquid_oxidation:

Reference: Execution time (DD:HH:MM:SS): 00:00:02:47
Current: Execution time (DD:HH:MM:SS): 00:00:03:14
Reference: Memory used: 2216.73 MB
Current: Memory used: 2210.45 MB

liquid_oxidation Failed Core Comparison ❌

Original model has 37 species.
Test model has 37 species. ✅
Original model has 216 reactions.
Test model has 215 reactions. ❌
The original model has 1 reactions that the tested model does not have. ❌
rxn: CCO[O](31) <=> [OH](22) + CC=O(69) origin: intra_H_migration

liquid_oxidation Failed Edge Comparison ❌

Original model has 202 species.
Test model has 202 species. ✅
Original model has 1610 reactions.
Test model has 1610 reactions. ✅
The original model has 1 reactions that the tested model does not have. ❌
rxn: CCO[O](31) <=> [OH](22) + CC=O(69) origin: intra_H_migration
The tested model has 1 reactions that the original model does not have. ❌
rxn: CCO[O](30) <=> C[CH]OO(70) origin: intra_H_migration

Observables Test Case: liquid_oxidation Comparison

✅ All Observables varied by less than 0.100 on average between old model and new model in all conditions!

liquid_oxidation Passed Observable Testing ✅

Regression test nitrogen:

Reference: Execution time (DD:HH:MM:SS): 00:00:01:44
Current: Execution time (DD:HH:MM:SS): 00:00:02:09
Reference: Memory used: 2214.09 MB
Current: Memory used: 2221.65 MB

nitrogen Passed Core Comparison ✅

Original model has 41 species.
Test model has 41 species. ✅
Original model has 360 reactions.
Test model has 360 reactions. ✅

nitrogen Passed Edge Comparison ✅

Original model has 132 species.
Test model has 132 species. ✅
Original model has 997 reactions.
Test model has 997 reactions. ✅

Observables Test Case: NC Comparison

✅ All Observables varied by less than 0.200 on average between old model and new model in all conditions!

nitrogen Passed Observable Testing ✅

Regression test oxidation:

Reference: Execution time (DD:HH:MM:SS): 00:00:02:50
Current: Execution time (DD:HH:MM:SS): 00:00:03:13
Reference: Memory used: 2084.02 MB
Current: Memory used: 2070.58 MB

oxidation Passed Core Comparison ✅

Original model has 59 species.
Test model has 59 species. ✅
Original model has 694 reactions.
Test model has 694 reactions. ✅

oxidation Passed Edge Comparison ✅

Original model has 230 species.
Test model has 230 species. ✅
Original model has 1526 reactions.
Test model has 1526 reactions. ✅

Observables Test Case: Oxidation Comparison

✅ All Observables varied by less than 0.500 on average between old model and new model in all conditions!

oxidation Passed Observable Testing ✅

Regression test sulfur:

Reference: Execution time (DD:HH:MM:SS): 00:00:01:08
Current: Execution time (DD:HH:MM:SS): 00:00:01:14
Reference: Memory used: 2183.30 MB
Current: Memory used: 2173.45 MB

sulfur Passed Core Comparison ✅

Original model has 27 species.
Test model has 27 species. ✅
Original model has 74 reactions.
Test model has 74 reactions. ✅

sulfur Failed Edge Comparison ❌

Original model has 89 species.
Test model has 89 species. ✅
Original model has 227 reactions.
Test model has 227 reactions. ✅
The original model has 1 reactions that the tested model does not have. ❌
rxn: O(4) + SO2(15) (+N2) <=> SO3(16) (+N2) origin: primarySulfurLibrary
The tested model has 1 reactions that the original model does not have. ❌
rxn: O(4) + SO2(15) (+N2) <=> SO3(16) (+N2) origin: primarySulfurLibrary

Observables Test Case: SO2 Comparison

The following observables did not match:

❌ Observable species O=S=O varied by more than 0.100 on average between old model SO2(15) and new model SO2(15) in condition 1.

⚠️ The following reaction conditions had some discrepancies:
Condition 1:
Reactor Type: IdealGasReactor
Reaction Time: 0.01 s
T0: 900 K
P0: 30 bar
Initial Mole Fractions: {'S': 0.000756, '[O][O]': 0.00129, 'N#N': 0.997954}

sulfur Failed Observable Testing ❌

Regression test superminimal:

Reference: Execution time (DD:HH:MM:SS): 00:00:00:43
Current: Execution time (DD:HH:MM:SS): 00:00:00:50
Reference: Memory used: 2322.90 MB
Current: Memory used: 2329.21 MB

superminimal Passed Core Comparison ✅

Original model has 13 species.
Test model has 13 species. ✅
Original model has 21 reactions.
Test model has 21 reactions. ✅

superminimal Failed Edge Comparison ❌

Original model has 18 species.
Test model has 18 species. ✅
Original model has 28 reactions.
Test model has 28 reactions. ✅

Non-identical kinetics! ❌
original:
rxn: O2(2) + [O]O(4) => [O]OO[O](17) + [H](3) origin: PDepNetwork #5
tested:
rxn: O2(2) + [O]O(4) => [O]OO[O](17) + [H](3) origin: PDepNetwork #5

k(1bar)	300K	400K	500K	600K	800K	1000K	1500K	2000K
k(T):	-83.19	-61.32	-48.07	-39.17	-27.88	-20.98	-11.59	-6.79
k(T):	-82.04	-60.38	-47.27	-38.46	-27.30	-20.50	-11.24	-6.53

kinetics: Chebyshev(coeffs=[[-39.45,-4.002e-05,-2.786e-05,-1.547e-05],[38.1,-4.003e-05,-2.787e-05,-1.547e-05],[0.4317,-1.799e-05,-1.252e-05,-6.95e-06],[0.1078,-7.549e-06,-5.254e-06,-2.917e-06],[0.02697,-3.017e-06,-2.1e-06,-1.166e-06],[-0.006763,-1.18e-06,-8.21e-07,-4.557e-07]], kunits='cm^3/(mol*s)', Tmin=(300,'K'), Tmax=(2000,'K'), Pmin=(0.01,'atm'), Pmax=(98.692,'atm'))
kinetics: Chebyshev(coeffs=[[-38.7,-3.696e-05,-2.573e-05,-1.429e-05],[37.66,-3.807e-05,-2.65e-05,-1.471e-05],[0.396,-1.752e-05,-1.219e-05,-6.77e-06],[0.09623,-7.467e-06,-5.197e-06,-2.885e-06],[0.01887,-2.976e-06,-2.072e-06,-1.15e-06],[-0.003474,-1.187e-06,-8.261e-07,-4.586e-07]], kunits='cm^3/(mol*s)', Tmin=(300,'K'), Tmax=(2000,'K'), Pmin=(0.01,'atm'), Pmax=(98.692,'atm'))
Identical kinetics comments:
kinetics:

Regression test RMS_constantVIdealGasReactor_superminimal:

Reference: Execution time (DD:HH:MM:SS): 00:00:03:21
Current: Execution time (DD:HH:MM:SS): 00:00:03:31
Reference: Memory used: 2753.11 MB
Current: Memory used: 2719.65 MB

RMS_constantVIdealGasReactor_superminimal Passed Core Comparison ✅

Original model has 13 species.
Test model has 13 species. ✅
Original model has 19 reactions.
Test model has 19 reactions. ✅

RMS_constantVIdealGasReactor_superminimal Passed Edge Comparison ✅

Original model has 13 species.
Test model has 13 species. ✅
Original model has 19 reactions.
Test model has 19 reactions. ✅

Observables Test Case: RMS_constantVIdealGasReactor_superminimal Comparison

✅ All Observables varied by less than 0.100 on average between old model and new model in all conditions!

RMS_constantVIdealGasReactor_superminimal Passed Observable Testing ✅

Regression test RMS_CSTR_liquid_oxidation:

Reference: Execution time (DD:HH:MM:SS): 00:00:08:16
Current: Execution time (DD:HH:MM:SS): 00:00:08:16
Reference: Memory used: 2712.73 MB
Current: Memory used: 2694.46 MB

RMS_CSTR_liquid_oxidation Failed Core Comparison ❌

Original model has 37 species.
Test model has 37 species. ✅
Original model has 232 reactions.
Test model has 233 reactions. ❌
The tested model has 1 reactions that the original model does not have. ❌
rxn: CCO[O](36) <=> [OH](22) + CC=O(61) origin: intra_H_migration

RMS_CSTR_liquid_oxidation Failed Edge Comparison ❌

Original model has 206 species.
Test model has 206 species. ✅
Original model has 1508 reactions.
Test model has 1508 reactions. ✅
The original model has 1 reactions that the tested model does not have. ❌
rxn: CCO[O](34) <=> C[CH]OO(62) origin: intra_H_migration
The tested model has 1 reactions that the original model does not have. ❌
rxn: CCO[O](36) <=> [OH](22) + CC=O(61) origin: intra_H_migration

Observables Test Case: RMS_CSTR_liquid_oxidation Comparison

✅ All Observables varied by less than 0.100 on average between old model and new model in all conditions!

RMS_CSTR_liquid_oxidation Passed Observable Testing ✅

beep boop this comment was written by a bot 🤖

rwest added 5 commits July 23, 2023 22:56

Only run annotation workflow on pull requests.

df61289

Hoping this works. Also fix a problem with escaping ' inside JSON when logging the github.event.workflow_run context.

Give the annotation runs a better name.

2e1c05d

Just skip annotation runs on failed pull requests.

6147eeb

Rather than run a pointless "the workflow failed" job, we just skip it.

rwest requested a review from JacksonBurns July 24, 2023 17:49

JacksonBurns approved these changes Jul 24, 2023

View reviewed changes

rwest merged commit eb178e1 into ReactionMechanismGenerator:main Jul 24, 2023
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More github workflow improvements #2506

More github workflow improvements #2506

rwest commented Jul 24, 2023

JacksonBurns left a comment

github-actions bot commented Jul 24, 2023

Regression test aromatics:

Regression test liquid_oxidation:

Regression test nitrogen:

Regression test oxidation:

Regression test sulfur:

Regression test superminimal:

Regression test RMS_constantVIdealGasReactor_superminimal:

Regression test RMS_CSTR_liquid_oxidation:

More github workflow improvements #2506

More github workflow improvements #2506

Conversation

rwest commented Jul 24, 2023

Motivation or Problem

Description of Changes

Testing

JacksonBurns left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 24, 2023

Regression Testing Results

Regression test aromatics:

Regression test liquid_oxidation:

Regression test nitrogen:

Regression test oxidation:

Regression test sulfur:

Regression test superminimal:

Regression test RMS_constantVIdealGasReactor_superminimal:

Regression test RMS_CSTR_liquid_oxidation: