A comprehensive, distributed, and fully automated test suite for the Ames Stereo Pipeline (ASP)
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
bin
ssDG_alignAffEpp_seedMode1_mapProj0_sgm
ssDG_alignAffEpp_seedMode1_mapProj0_subPix1
ssDG_alignHom_seedMode1_mapProj0_subPix1
ssDG_alignHom_seedMode1_mapProj0_subPix1_BasaltHills
ssDG_alignHom_seedMode1_mapProj0_subPix1_BasaltHills_fillHoles
ssDG_alignHom_seedMode1_mapProj0_subPix1_BasaltHills_jitter
ssDG_alignHom_seedMode1_mapProj0_subPix1_BasaltHills_multi_parallel
ssDG_alignHom_seedMode1_mapProj0_subPix1_BasaltHills_smallDem
ssDG_alignHom_seedMode1_mapProj0_subPix1_localHom
ssDG_alignNone_seedMode1_mapProj1_subPix1_BasaltHills
ssDG_alignNone_seedMode1_mapProj1_subPix1_badDisp1
ssDG_alignNone_seedMode1_mapProj1_subPix1_leastSq
ssDG_alignNone_seedMode1_mapProj1_subPix1_no_default
ssDG_alignNone_seedMode1_mapProj1_subPix2
ssDG_alignNone_seedMode1_mapProj1_subPix2_leastSq
ssDG_alignNone_seedMode2_mapProj0_subPix1_localHom
ssDG_alignNone_seedMode2_mapProj0_subPix2_localHom
ssDG_alignNone_seedMode3_mapProj1_subPix1_badDisp1
ssDG_alignNone_seedMode3_mapProj1_subPix1_badDisp2
ssDG_alignNone_seedMode3_mapProj1_subPix1_parallel
ssISISMAPISIS_alignHom_seedMode1_mapProj0_subPix1
ssISIS_alignAffEp_seedMode1_mapProj0_subPix1_frameCam
ssISIS_alignAffEpp_seedMode1_mapProj0_subPix1
ssISIS_alignAffEpp_seedMode1_mapProj0_subPix1_parallel
ssISIS_alignAffEpp_seedMode2_mapProj0_SGM
ssISIS_alignAffEpp_seedMode3_mapProj0_subPix1
ssISIS_alignHom_seedMode1_mapProj0_subPix1
ssISIS_alignHom_seedMode1_mapProj0_subPix1_crop_left_right
ssISIS_alignHom_seedMode1_mapProj0_subPix1_localHom
ssISIS_alignHom_seedMode1_mapProj0_subPix1_nocrop
ssISIS_alignHom_seedMode1_mapProj0_subPix1_optThresh
ssISIS_alignHom_seedMode1_mapProj0_subPix1_parallel
ssISIS_alignHom_seedMode1_mapProj1_subPix1_flyby
ssISIS_alignHom_seedMode2_mapProj0_subPix1
ssISIS_alignHom_seedMode2_mapProj0_subPix1_localHom
ssISIS_alignHom_seedMode3_mapProj0_subPix1
ssISIS_alignNone_seedMode1_mapProj1_subPix1
ssISIS_alignNone_seedMode1_mapProj1_subPix1_pluto
ssISIS_alignNone_seedMode2_mapProj0_subPix1_localHom
ssNadirPinHole_alignEpi_seedMode1_mapProj0_subPix1_OIB_dist
ssNadirPinHole_alignHom_seedMode1_mapProj0_subPix1
ssNadirPinHole_alignHom_seedMode1_mapProj0_subPix1_Earth
ssPinHole_alignEpi_seedMode1_mapProj0_subPix1_OIB_dist
ssPinHole_alignEpipolar_seedMode1_mapProj0_subPix1
ssPinHole_alignHom_seedMode0_mapProj0_subPix1_MER
ssPinHole_alignHom_seedMode1_mapProj0_subPix1
ssPinHole_alignHom_seedMode1_mapProj0_subPix1_Earth
ssPinHole_alignHom_seedMode1_mapProj0_subPix1_parallel1
ssPinHole_alignHom_seedMode1_mapProj0_subPix1_parallel2
ssPinHole_alignNone_seedMode1_mapProj1_subPix1
ssRPC_alignAffep_Mapproj0_Mars
ssRPC_alignHom_seedMode1_mapProj0_subPix1_BasaltHills
ssRPC_alignNone_mapProj1_Cartosat
ssRPC_alignNone_seedMode0_mapProj0_subPix1
ssRPC_alignNone_seedMode1_mapProj0_subPix1
ssRPC_alignNone_seedMode1_mapProj0_subPix2_localHom
ssRPC_alignNone_seedMode1_mapProj1_subPix1_BasaltHills
ss_aster2asp
ss_bundle_adjust_brownconrady
ss_bundle_adjust_dg_ceres
ss_bundle_adjust_filter_outliers
ss_bundle_adjust_fix_one_cam
ss_bundle_adjust_isis_ceres
ss_bundle_adjust_isis_robust_sparse
ss_bundle_adjust_null
ss_bundle_adjust_photometrix
ss_bundle_adjust_rpc_mars
ss_bundle_adjust_terrain_constraint
ss_bundle_adjust_tsai
ss_cam2map4stereo
ss_cam2rpc_mars
ss_camera_calibrate
ss_camera_solve_mt_sidley
ss_colormap
ss_convert_pinhole_model
ss_datum_convert
ss_dem_geoid_egm2008
ss_dem_geoid_egm96
ss_dem_geoid_mola
ss_dem_geoid_mola_diffHeight
ss_dem_geoid_navd88
ss_dem_mosaic_360ofset
ss_dem_mosaic_centerline
ss_dem_mosaic_first_last
ss_dem_mosaic_lonlat
ss_dem_mosaic_lonlat_utm
ss_dem_mosaic_nooverlap
ss_dem_mosaic_nooverlap2
ss_dem_mosaic_pluto
ss_dem_mosaic_polar
ss_dem_mosaic_priority
ss_dem_mosaic_single
ss_dem_mosaic_unsmooth_wt
ss_dem_mosaic_utm_lonlat
ss_dg_mapproject_bigdem
ss_dg_mosaic
ss_dg_mosaic_180bug
ss_dg_mosaic_180bug2
ss_dg_mosaic_llboxbug
ss_dg_mosaic_llboxbug2
ss_dg_mosaic_llboxbug3
ss_dg_mosaic_multispectral
ss_dg_mosaic_nodata
ss_dg_mosaic_rpc_gen
ss_dg_mosaic_var_pitch
ss_geodiff
ss_geodiff_360offset
ss_geodiff_bug
ss_geodiff_csv
ss_hiedr2mosaic
ss_hillshade
ss_image_calc
ss_isis_mapproject_lroc_gsd
ss_isis_mapproject_moc
ss_lronac2mosaic
ss_mapproject_ctx_bug
ss_mapproject_isis
ss_mapproject_tr_bug
ss_n_align_csv
ss_orbitviz_earth
ss_orbitviz_moon
ss_pansharp
ss_parallel_sfs
ss_pc_align_SanLuisResevoir
ss_pc_align_alaska
ss_pc_align_alaska_dem_dem
ss_pc_align_alaska_z_lat_lon
ss_pc_align_charon
ss_pc_align_dem2dem_bug1
ss_pc_align_dem2dem_bug2
ss_pc_align_fgr
ss_pc_align_filled_SanLuisResevoir
ss_pc_align_hillshade
ss_pc_align_large_shift
ss_pc_align_las
ss_pc_align_least_sq
ss_pc_align_lola_radius_km
ss_pc_align_lola_radius_m
ss_pc_align_lola_rdr
ss_pc_align_lola_rdr_p2point
ss_pc_align_lola_rdr_similarity
ss_pc_align_lola_rdr_small
ss_pc_align_manual_trans
ss_pc_align_mars_dem_csv
ss_pc_align_mars_dt4
ss_pc_align_mars_pc_csv
ss_pc_align_mars_pc_csv_neglon
ss_pc_align_mars_pc_pc
ss_pc_align_mars_pc_xyz
ss_pc_align_point_to_plane
ss_pc_align_proj4
ss_pc_align_proj4_csv_csv
ss_pc_align_similarity
ss_pc_align_similarity_least_sq
ss_pc_align_similarity_point_to_point
ss_pc_align_utm
ss_pc_merge
ss_pinhole_mapproject_gray
ss_pinhole_mapproject_rgb
ss_point2dem_csv_lat_lon_height
ss_point2dem_csv_lat_lon_radius
ss_point2dem_csv_proj4
ss_point2dem_csv_reproj
ss_point2dem_csv_utm
ss_point2dem_csv_xyz
ss_point2dem_las
ss_point2dem_las_no_georef
ss_point2dem_nad83
ss_point2dem_nz
ss_point2las_nad83
ss_point2las_nozip
ss_point2las_srs
ss_point2las_zip
ss_point2mesh
ss_rpc_mapproject_bigdem
ss_rpc_mapproject_cambox_bug
ss_rpc_mapproject_cambox_bug2
ss_rpc_mapproject_small_dem
ss_rpc_mapproject_unitDegree
ss_sfs
ss_stereo_spot5
ss_undistort_image
ss_wv_correct
ss_wv_correct_wv01_fwd
.gitignore
README.txt
conftest.py
lunokhod1_scott.conf
pfe.conf
release_amos.conf
release_andey.conf
release_centos-6.0.conf
release_centos-6.conf
release_decoder.conf
release_lunokhod1.conf
release_lunokhod2.conf
test_run.py

README.txt

StereoPipelineTest is a comprehensive, distributed, and fully
automated test suite for the Ames Stereo Pipeline (ASP). It attempts
to cover most, if not all, of the ways in which ASP can be used, and
the test suite should be updated regularly as more functionality is
added to ASP.

StereoPipelineTest uses the Python py.test framework to run the tests.
This framework needs Python >= 2.6. It can be installed as follows:

pip install --user pytest
pip install --user pytest-xdist

or 

easy_install --prefix ~/.local -U pytest
easy_install --prefix ~/.local -U pytest-xdist

The py.test script ends up being installed either in /usr/bin, or in
~/.local/bin (on Linux) or in ~/Library/Python/2.x/bin (on OSX). In
either case, the install directory needs to be added to the path.

Usage: py.test -n <num cpu> -q -s -r a --tb=no --config <settings file> > report.txt

Here, <num cpu> is how many processes to use. 

The files conftest.py and test_run.py control the behavior of py.test.

A sample <settings file> is provided, named 'pfe.conf'. This file has:

1. The tests to run (wildcard expressions are accepted).
2. The machines to distribute the runs across (they must be accessible
   via ssh and share disk storage).
3. How many processes to use on each machine (each process in turn uses 
   multiple threads).
4. If to do strict validation (that is, not allow, vs. allow, a small
   discrepancy between current and reference runs, more below).
5. Environmental variables, such as the path to the ASP executables.

Each test needs to be in its own directory. A test is executed by
running the script 'run.sh' in that directory, which should create an
output directory named 'run'. A 'gold' directory must be present,
which has the reference run. At the conclusion of the run, the result in
the 'run' directory is compared to the reference in 'gold'. The test
will fail if the produced result is different or absent.

If a new test is added, the name of the test directory must be listed
in the settings file or match the wildcard pattern already present
there. Each test must have a 'run.sh' file, and a validation script,
named 'validate.sh'. The 'validate.sh' script must return exit status
0 on successful validation, and non-zero otherwise.

When tests fail, which is inevitable when something changes, and the
new results are deemed acceptable, the 'gold' reference directory
needs to be updated by copying the output from the 'run' directory. 

If the processes is terminated before all tests finish, the last result
for each test can be seen by running bin/check_status.py.

--== Troubleshooting ==--

If py.test is not found, try running 
source [BINARY_BUILDER_DIR]/auto_build/utils.sh

--== Tools ==--

There are a number of utilties in the main and bin/ folders which can
be useful for working with tests.  These include:

runs_to_golds.py = Convert all run folders to gold folders or pass in a
single test name to just convert the folders for that test.

check_status.py = Tries to print the last run status of each test.  Use
the "--conf-file" argument to limit the list to non-disabled tests.