Not all tests are run in Azure's Python builds

[fbudin69500]

Description

Not all Python tests are run on Azure's Python Build (42 instead of 81).

Steps to Reproduce

To check what Python tests are run in Azure's Python's builds, one can connect to dev.azure.com from the check result links on Github. If the PR has been closed, one can still access the check result links by clicking on the green check mark or red cross to expand the list of checks. For example, for PR #268 , the results of the Linux Python Builds are available here.

On the Build and test line, one can click on the blue icon download the log to download a zip file containing the build and test log.

After that, the following command can be run to count the number of Python tests that were run:
more 7.txt|grep Python |grep Test |grep '#' |wc -l

Expected behavior

In theory, there should be around 80 tests run for Python builds, based on what happens on my machine.

Actual behavior

Only 42 Python tests are run.

Reproducibility

Unknown as the PR used to reproduce this step was the first re-activating about 40 tests that had been accidently deactivated.

Versions

At least for PR #268

Environment

I haven't checked the logs on Max and Windows, but I expect the same behavior since all the tests were reporting as passing even though there should have been some tests failing. This demonstrates that the tests that should have failed were not run.

[fbudin69500]

Confirming with PR #272 that this is a problem: Only 42 tests instead of 81.

[thewtex]

To confirm, the Python builds for PR #268 are here.

• Linux has 42 "Python" tests
• macOS has 44 "Python" tests
• Windows has 11 "Python" tests

The fewer tests on Windows is expected because we are only testing Group Core due to build time.

Locally, with ctest -R Python -N, I get 44 tests. What is the output on your system with 80 tests?

[fbudin69500]

Is your local build more recent than 9d47bda?

[fbudin69500]

On my computer (Linux build), for the following commit:

git log -1
commit baf3389676a3beb7dad482eb3f244d0392bb61d5 (HEAD -> master, upstream/python-builds,     upstream/nightly-master, upstream/master)
Merge: 3f395b1dc1 9f2e754ac8
Author: Dženan Zukić <dzenan.zukic@kitware.com>
Date:   Wed Dec 5 10:13:56 2018 -0500

Merge pull request #261 from dzenanz/vsConformanceMode

COMP: compile with Visual Studio in conformance mode AKA /permissive-`

I get:

Test  #179: PythonCurvatureAnisotropicDiffusionImageFilterTest
Test  #180: PythonSmoothingRecursiveGaussianImageFilterTest
Test  #181: PythonGradientAnisotropicDiffusionImageFilterTest
Test  #229: PythonAntiAliasBinaryImageFilterTest
Test  #265: PythonBinaryDilateImageFilterTest
Test  #266: PythonBinaryErodeImageFilterTest
Test  #567: itkMetaDataDictionaryPythonTest
Test  #568: itkDirectoryPythonTest
Test  #569: itkObjectPythonTest
Test  #570: itkIndexOffsetPythonTest
Test  #624: PythonCurvatureFlowImageFilterTest
Test  #744: PythonFFTImageFilterTest
Test  #776: PythonFastMarchingLeftVentricleTest
Test  #777: PythonFastMarchingRightVentricleTest
Test  #778: PythonFastMarchingWhiteMatterTest
Test  #779: PythonFastMarchingGrayMatterTest
Test  #842: PythonReadDicomAndReadTagTest
Test #1328: itkGradientVectorFlowImageFilterPythonTest
Test #1329: PythonLaplacianImageFilterTest
Test #1330: PythonCannyEdgeDetectionImageFilterTest
Test #1340: PythonCastImageFilterTest
Test #1350: itkUnaryFrequencyDomainFilterPythonTest
Test #1351: itkFrequencyBandImageFilterPythonTest
Test #1419: PythonGradientMagnitudeRecursiveGaussianImageFilterTest
Test #1500: PythonResampleImageFilterTest1
Test #1501: PythonResampleImageFilterTest2
Test #1502: PythonResampleImageFilterTest3
Test #1503: PythonResampleImageFilterTest4
Test #1591: PythonSigmoidImageFilterTest
Test #1835: PythonLazyLoadingImage
Test #1836: PythonGeodesicActiveContourLeftVentricleTest
Test #1837: PythonThresholdSegmentationLevelSetVentricleTest
Test #1838: PythonThresholdSegmentationLevelSetWhiteMatterTest
Test #1839: PythonThresholdSegmentationLevelSetGrayMatterTest
Test #1954: PythonGrayscaleDilateImageFilterTest
Test #1955: PythonGrayscaleErodeImageFilterTest
Test #1956: PythonBoxGrayscaleDilateImageFilterTest
Test #2012: itkBYUMeshIOPythonTest
Test #2013: itkFreeSurferAsciiMeshIOPythonTest
Test #2014: itkFreeSurferBinaryMeshIOPythonTest
Test #2015: itkMeshFileReaderPythonTest
Test #2016: itkMeshFileWriterPythonTest
Test #2017: itkOBJMeshIOPythonTest
Test #2018: itkOFFMeshIOPythonTest
Test #2019: itkVTKPolyDataMeshIOPythonTest
Test #2060: itkDemonsImageToImageMetricv4PythonTest
Test #2061: itkCorrelationImageToImageMetricv4PythonTest
Test #2062: itkJointHistogramMutualInformationImageToImageMetricv4PythonTest
Test #2142: itkParametricPathPythonTest
Test #2143: itkPolyLineParametricPathPythonTest
Test #2144: itkImageToPathPythonTest
Test #2145: itkPathToImageFilterPythonTest
Test #2347: PythonMeanImageFilterTest
Test #2348: PythonMeanImageFilterFunctionCallTest
Test #2349: PythonMedianImageFilterTest
Test #2350: PythonAutoPipelineTest
Test #2384: PythonSpatialObjectTest
Test #2492: itkNormalVariateGeneratorPythonTest
Test #2562: PythonBinaryThresholdImageFilterTest
Test #2563: PythonThresholdImageFilterTest
Test #2664: PythonWatershedSegmentation1Test
Test #2665: NonBlockingPythonFilterCoverage
Test #2666: NonBlockingPythonSpatialObjectCoverage
Test #2667: NonBlockingPythonCostFunctionCoverage
Test #2668: PythonOptimizerCoverage
Test #2669: PythonImageIOCoverage
Test #2670: PythonTransformCoverage
Test #2671: PythonTiming
Test #2672: PythonComplex
Test #2673: PythonTypemapsTest
Test #2674: PythonTemplateTest
Test #2675: PythonModifiedTimeTest
Test #2676: PythonTemplatedPipelineTest
Test #2677: PythonModuleToModuleTest
Test #2678: PythonExtrasTest
Test #2679: PythonSimplePipelineSS2
Test #2680: PythonSimplePipelineUC2
Test #2681: PythonSimplePipelineF2
Test #2682: PythonSimplePipelineSS3
Test #2683: PythonSimplePipelineUC3
Test #2684: PythonSimplePipelineF3

Total Tests: 81

[fbudin69500]

FYI: I was just thinking that I should check the dashboards and the Python nighly builds to see if all the Python tests where running on there. But something might be broken there as only the Mac build is done, but fails. The linux and Windows builds do not seem to return any result (at least for yesterday, 12/06/18). I thought I would report this issue here as it may be related to the original problem I reported.

[thewtex]

I confirmed that I could reproduce the 81 tests with a local build.

The macOS Nightly Python build is also only reporting 45 tests. We'll eventually bring the Linux and Windows build back online.

I suspect there is a CMake variable caching or scope issue that is revealed by running configuration through the CTest script...

[thewtex]

Update: I was able to reproduce locally running via CTest script. With setting dashboard_no_clean and running again, the tests appear, and some of them fail:

https://open.cdash.org/index.php?project=Insight&filtercount=1&showfilters=1&field1=site&compare1=63&value1=clay

[fbudin69500]

Thanks for solving that mystery. The failing tests might be related to #271, but you can double check that it addresses all the issues listed on the dashboard.