Move large tests into main Jenkinsfile #2056

bilke · 2018-01-25T14:17:51Z

WIP!

Idea: Move large tests into main Jenkinsfile and test if ufz/master runs. Then it is possible to check if all benchmarks added in CMake files with Add_Test() are really tested. This will not increase the overall run time of the Jenkins job as the large tests are conducted on another machine (@endJunction I plan to setup another spare PC here to relieve your machine..).

Related: #2053, #2054, #2055.

endJunction · 2018-01-25T16:12:50Z

Nice! For the time being, you can disable the failing large tests, s.t. we have a stable state.

bilke · 2018-01-25T19:10:19Z

With OGS_USE_LIS, short & large benchmarks:

The following tests FAILED:
	 18 - ogs-LARGE_2D_ComponentTransport_Elder-time (Failed)
	 19 - ogs-LARGE_2D_ComponentTransport_Elder-time-vtkdiff (Failed)
	 21 - ogs-2D_ComponentTransport_HeterogeneousPermeability-time-vtkdiff (Failed)
	162 - ogs-HydroMechanics_HML_square_1e2_unconfined_compression_early-time (Failed)
	163 - ogs-HydroMechanics_HML_square_1e2_unconfined_compression_early-time-vtkdiff (Failed)
	166 - ogs-MonolithicInjectionProduction1D-time (Failed)
	167 - ogs-MonolithicInjectionProduction1D-time-vtkdiff (Failed)
	202 - ogs-RichardsComponentTransport_1D_Padilla_NaCl1-time (Failed)
	203 - ogs-RichardsComponentTransport_1D_Padilla_NaCl1-time-vtkdiff (Failed)
	204 - ogs-RichardsComponentTransport_1D_Padilla_NaCl6-time (Failed)
	205 - ogs-RichardsComponentTransport_1D_Padilla_NaCl6-time-vtkdiff (Failed)
	213 - ogs-LARGE_Mechanics_SDL_square_1e5_tractionBC-vtkdiff (Failed)
	215 - ogs-Mechanics_SDL_square_1e2_quad8_traction_topBC-time-vtkdiff (Failed)
	238 - ogs-LARGE_SmallDeformation_PressureBC_hollow_sphere (Failed)
	239 - ogs-LARGE_SmallDeformation_PressureBC_hollow_sphere-vtkdiff (Failed)
	245 - ogs-SmallDeformation_PressureBC_elastic_sphere_axisymmetric-vtkdiff (Failed)
	246 - ogs-SmallDeformation_PressureBC_plastic_sphere_axisymmetric (Failed)
	247 - ogs-SmallDeformation_PressureBC_plastic_sphere_axisymmetric-vtkdiff (Failed)
	249 - ogs-SmallDeformation_SDL_two_material_gravity-vtkdiff (Failed)

With LIS off and large only:

The following tests FAILED:
169 - ogs-LARGE_2D_ThermalConvection_constviscosity-time-vtkdiff (Failed)
171 - ogs-LARGE_2D_ThermalConvection_constviscosityStaggeredScheme-time-vtkdiff (Failed)
239 - ogs-LARGE_Mechanics_SDL_square_1e5_tractionBC-vtkdiff (Failed)
285 - ogs-LARGE_2D_ThermoElastic_IGLU_Plane_Strain_Quadratic_Mesh-time-vtkdiff (Failed)
300 - ogs-LARGE_2D_TwoPhase_Prho_MoMaS (Failed)
301 - ogs-LARGE_2D_TwoPhase_Prho_MoMaS-vtkdiff (Failed)

I think we should focus on these first. Later we can (re-) introduce LIS tests again as a separate config run.

endJunction · 2018-01-26T00:04:58Z

I'll fix the mechanics benchmarks.
Tom and Marc please have a look into the component transport and thermal convection.
Who is taking the Richards and injection tests?

bilke · 2018-01-26T07:01:24Z

Please note that you can now see the output of failing benchmarks (was not shown before). E.g.:

https://jenkins.opengeosys.org/job/ufz/job/ogs/job/PR-2056/5/console

endJunction · 2018-01-29T15:04:39Z

@bilke One can choose shared libraries for the large tests build... Saves a little time.

bilke · 2018-02-02T13:58:38Z

Don't know if the result with this large absolute error makes sense: 6ab4209

chleh · 2018-02-12T11:18:47Z

Don't know if the result with this large absolute error makes sense: 6ab4209

@bilke AFAIK the test will path if at least one of the abs or rel tolerances are fulfilled. So in principle you could specify 0 1e-2, but still one might check if that large relative tolerance makes sense. Furthermore, of there are some zero pressure values, the relative tolerance might always fail.

bilke · 2018-03-16T07:42:59Z

Rebased. Following benchmarks are still failing:

LARGE_2D_TwoPhase_Prho_MoMaS:

  info: *** Eigen solver computation

  info: -> solve with BiCGSTAB (precon ILUT)

  info: 	 iteration: 10000/10000

  info: 	 residual: 1.134419e+11

  

  error: Failed during Eigen linear solve

  info: ------------------------------------------------------------------

  info: [time] Linear solver took 130.124 s.

  error: Newton: The linear solver failed.

  info: [time] Solving process #0 took 130.976 s in time step #1

  error: The nonlinear solver failed in time step #1 at t = 1000 s for
  process #0.

  info: [time] Output of timestep 1 took 0.00803518 s.

  error: Nonlinear solver fails.  Because of the time stepper of
  FixedTimeStepping is used, the program has to be terminated at
  UncoupledProcessesTimeLoop.cpp, line 696

LARGE_GroundWaterFlowProcess_square_1x1_1e6:

Computed difference between data arrays:

  abs l1 norm = 5.279337400309042e+05

  abs l2-norm^2 = 3.578798701390355e+05

  abs l2-norm = 5.982306161832872e+02

  abs maximum norm = 1.401070486487338e+00

  

  rel l1 norm = inf

  rel l2-norm^2 = inf

  rel l2-norm = inf

  rel maximum norm = inf

  Absolute and relative error (maximum norm) are larger than the
  corresponding thresholds.

LARGE_PhaseField_Staggered_square_line:

Error reading file
233:   `..../build-n/Tests/Data/PhaseField/square_line_h_400_pcs_0_ts_400000_t_400.000000.vtu'.
...
$ ls build-n/Tests/Data/PhaseField/square_line_h_400_pcs_0_ts_*
build-n/Tests/Data/PhaseField/square_line_h_400_pcs_0_ts_0_t_0.000000.vtu
build-n/Tests/Data/PhaseField/square_line_h_400_pcs_0_ts_10000_t_1.000000.vtu
build-n/Tests/Data/PhaseField/square_line_h_400_pcs_0_ts_2000_t_0.200000.vtu
build-n/Tests/Data/PhaseField/square_line_h_400_pcs_0_ts_4000_t_0.400000.vtu
build-n/Tests/Data/PhaseField/square_line_h_400_pcs_0_ts_6000_t_0.600000.vtu
build-n/Tests/Data/PhaseField/square_line_h_400_pcs_0_ts_8000_t_0.800000.vtu

# timesteps outputted are different!

LARGE_PhaseField_Staggered_square_shear: same problem as 3.
LiquidFlow_BuildupTest (Appveyor only, our own Windows build is fine):

  Computed difference between data arrays:
7234
7235  abs l1 norm = 1.216519682884609e-03
7236
7237  abs l2-norm^2 = 2.299775509368990e-09
7238
7239  abs l2-norm = 4.795597469939475e-05
7240
7241  abs maximum norm = 2.555083483457565e-06
7242
7243  
7244
7245  rel l1 norm = 2.051569512828749e-09
7246
7247  rel l2-norm^2 = 5.114652784891431e-19
7248
7249  rel l2-norm = 7.151680071767354e-10
7250
7251  rel maximum norm = 6.907157957013411e-10
7252
7253  Absolute and relative error (maximum norm) are larger than the
7254  corresponding thresholds.
7255
7256
7257

Compare to the original reference result for t=300 and ts=1e-2, which matched sufficiently good. Increase the time step size to 1e-1 and increase the end time to 400. New results included. Be aware, that the test result is wrong and is there for code coverage.

The original reference result was reproduced. New result for shorter time is provided.

Also remove some of the ctests from LARGE if running fast according to the last jenkins' output.

codecov · 2019-02-01T12:43:29Z

Codecov Report

Merging #2056 into master will increase coverage by 0.46%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2056      +/-   ##
==========================================
+ Coverage   32.43%   32.89%   +0.46%     
==========================================
  Files         526      525       -1     
  Lines       20016    19720     -296     
  Branches     4051     4050       -1     
==========================================
- Hits         6492     6487       -5     
+ Misses      10281     9989     -292     
- Partials     3243     3244       +1

Impacted Files	Coverage Δ
GeoLib/SurfaceGrid.cpp	`53.7% <0%> (-2.78%)`	⬇️
GeoLib/Polyline.cpp	`35.75% <0%> (-0.61%)`	⬇️
...ousMedium/Permeability/createPermeabilityModel.cpp	`0% <0%> (ø)`	⬆️
Applications/ApplicationsLib/ProjectData.cpp	`0% <0%> (ø)`	⬆️
...Lib/PorousMedium/Permeability/DupuitPermeability.h

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ca0fe1...7a55b0c. Read the comment docs.

endJunction · 2019-02-01T23:08:41Z

@bilke Looks like all tests (even the large tests) passed, but still that particular node is marked red.

TomFischer · 2019-02-06T06:51:26Z

Nice, thanks for your work!

ogsbot · 2020-06-19T12:51:01Z

OpenGeoSys development has been moved to GitLab.

See this pull request on GitLab.

bilke added the WIP 🏗️ label Jan 25, 2018

bilke force-pushed the move-large-tests branch from 7750bb6 to 78d2504 Compare January 25, 2018 19:21

endJunction force-pushed the move-large-tests branch 2 times, most recently from f084864 to 594fbd1 Compare January 29, 2018 09:56

bilke force-pushed the move-large-tests branch from 1aaeffa to 2754205 Compare February 2, 2018 10:42

bilke force-pushed the move-large-tests branch 3 times, most recently from 8513b46 to 717630c Compare March 15, 2018 14:41

endJunction force-pushed the move-large-tests branch from c39a9ad to c39e0fd Compare May 7, 2018 21:05

endJunction force-pushed the move-large-tests branch from c39e0fd to 8d46fa7 Compare May 25, 2018 14:02

endJunction force-pushed the move-large-tests branch from 1d4421b to c0f8413 Compare June 19, 2018 15:13

wenqing mentioned this pull request Jun 20, 2018

[Test] Changed the linear solver type in Parabolic/TwoPhaseFlowPrho/MoMaS/Twophase_MoMaS_quad.prj #2148

Closed

endJunction mentioned this pull request Jun 28, 2018

Fix some ctests #2155

Merged

endJunction force-pushed the move-large-tests branch from c0f8413 to bfb17f9 Compare June 29, 2018 21:21

bilke force-pushed the master branch from 7e4d94d to 26445e2 Compare November 16, 2018 13:23

endJunction force-pushed the move-large-tests branch from bfb17f9 to 64044bb Compare January 7, 2019 10:40

endJunction force-pushed the move-large-tests branch 3 times, most recently from 310cc2d to 6c8a2e8 Compare January 20, 2019 23:43

endJunction force-pushed the move-large-tests branch 3 times, most recently from 43b12b3 to 32a33e3 Compare February 1, 2019 08:31

bilke and others added 13 commits February 1, 2019 13:42

[Jenkins] Moved large-tests into main Jenkinsfile.

4528d6e

[Jenkins] Test large benchmarks only, switch off LIS.

2ba9b9e

[CMake] Fixed error output when running ctest-targets.

e3fe113

[T] Relaxed tolerances for HT process large test.

fec9170

[T] Relaxed tolerance for pressure even more .

6c1a8bc

[ctest] Moved some benchmarks out of LARGE.

93066db

[T] Set Goswami and TM_IGLP plane test LARGE.

38f7492

[Jenkins] Migrate Tests-Large job to dockerfile-syntax.

463f299

[T] Relax few more of the large tests' tolerances.

417064e

[T] Fix variables output of some TMPf ctests.

bffbd08

[T] PF; Update line test.

6304642

The original reference result was reproduced. New result for shorter time is provided.

[T] Add RUNTIME to large tests.

c3775cb

Also remove some of the ctests from LARGE if running fast according to the last jenkins' output.

endJunction force-pushed the move-large-tests branch from 32a33e3 to c3775cb Compare February 1, 2019 12:43

endJunction added please review and removed WIP 🏗️ labels Feb 1, 2019

bilke added 2 commits February 4, 2019 11:26

[Jenkins] Fixed test result publishing.

1c3b4e0

[Jenkins] Moved large tests to master-only.

7a55b0c

bilke removed the please review label Feb 5, 2019

bilke merged commit 6d94d62 into ufz:master Feb 5, 2019

bilke deleted the move-large-tests branch February 5, 2019 14:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move large tests into main Jenkinsfile #2056

Move large tests into main Jenkinsfile #2056

bilke commented Jan 25, 2018 •

edited

Loading

endJunction commented Jan 25, 2018

bilke commented Jan 25, 2018 •

edited

Loading

endJunction commented Jan 26, 2018 •

edited

Loading

bilke commented Jan 26, 2018

endJunction commented Jan 29, 2018

bilke commented Feb 2, 2018 •

edited

Loading

chleh commented Feb 12, 2018 •

edited

Loading

bilke commented Mar 16, 2018 •

edited

Loading

codecov bot commented Feb 1, 2019 •

edited

Loading

endJunction commented Feb 1, 2019

TomFischer commented Feb 6, 2019

ogsbot commented Jun 19, 2020

Move large tests into main Jenkinsfile #2056

Move large tests into main Jenkinsfile #2056

Conversation

bilke commented Jan 25, 2018 • edited Loading

endJunction commented Jan 25, 2018

bilke commented Jan 25, 2018 • edited Loading

endJunction commented Jan 26, 2018 • edited Loading

bilke commented Jan 26, 2018

endJunction commented Jan 29, 2018

bilke commented Feb 2, 2018 • edited Loading

chleh commented Feb 12, 2018 • edited Loading

bilke commented Mar 16, 2018 • edited Loading

codecov bot commented Feb 1, 2019 • edited Loading

Codecov Report

endJunction commented Feb 1, 2019

TomFischer commented Feb 6, 2019

ogsbot commented Jun 19, 2020

bilke commented Jan 25, 2018 •

edited

Loading

bilke commented Jan 25, 2018 •

edited

Loading

endJunction commented Jan 26, 2018 •

edited

Loading

bilke commented Feb 2, 2018 •

edited

Loading

chleh commented Feb 12, 2018 •

edited

Loading

bilke commented Mar 16, 2018 •

edited

Loading

codecov bot commented Feb 1, 2019 •

edited

Loading