unittests failing @ NERSC from NIGHT/EXPID/rawdata location

[sbailey]

Nightly unit tests are failing at NERSC with the tracebacks below. These appear to be due to the reorganization of raw data into NIGHT/EXPID/ directories instead of just NIGHT/ directories, but I thought that was fixed in PR #384 . Assigning to 18.6 release since this need to be resolved before next major software release.

======================================================================
ERROR: test_quickgen_airmass_simspec (desisim.test.test_quickgen.TestQuickgen)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/test/test_quickgen.py", line 278, in test_quickgen_airmass_simspec
    quickgen.main(quickgen.parse(cmd.split()[1:]))
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/scripts/quickgen.py", line 211, in main
    fibermap=read_fibermap(args.fibermap)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/code/desispec/master/py/desispec/io/fibermap.py", line 147, in read_fibermap
    return Table.read(filename, 'FIBERMAP')
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/table/table.py", line 2519, in read
    out = io_registry.read(cls, *args, **kwargs)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/io/registry.py", line 516, in read
    fileobj = ctx.__enter__()
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/utils/data.py", line 208, in get_readable_fileobj
    fileobj = io.FileIO(name_or_obj, 'r')
FileNotFoundError: [Errno 2] No such file or directory: '/global/homes/d/desi/desi_test_io/spectro/sim/test-quickgen/20150105/fibermap-00000100.fits'

======================================================================
ERROR: test_quickgen_exptime_simspec (desisim.test.test_quickgen.TestQuickgen)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/test/test_quickgen.py", line 306, in test_quickgen_exptime_simspec
    quickgen.main(quickgen.parse(cmd.split()[1:]))
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/scripts/quickgen.py", line 211, in main
    fibermap=read_fibermap(args.fibermap)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/code/desispec/master/py/desispec/io/fibermap.py", line 147, in read_fibermap
    return Table.read(filename, 'FIBERMAP')
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/table/table.py", line 2519, in read
    out = io_registry.read(cls, *args, **kwargs)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/io/registry.py", line 516, in read
    fileobj = ctx.__enter__()
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/utils/data.py", line 208, in get_readable_fileobj
    fileobj = io.FileIO(name_or_obj, 'r')
FileNotFoundError: [Errno 2] No such file or directory: '/global/homes/d/desi/desi_test_io/spectro/sim/test-quickgen/20150105/fibermap-00000100.fits'

======================================================================
ERROR: test_quickgen_moonangle_simspec (desisim.test.test_quickgen.TestQuickgen)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/test/test_quickgen.py", line 362, in test_quickgen_moonangle_simspec
    quickgen.main(quickgen.parse(cmd.split()[1:]))
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/scripts/quickgen.py", line 211, in main
    fibermap=read_fibermap(args.fibermap)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/code/desispec/master/py/desispec/io/fibermap.py", line 147, in read_fibermap
    return Table.read(filename, 'FIBERMAP')
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/table/table.py", line 2519, in read
    out = io_registry.read(cls, *args, **kwargs)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/io/registry.py", line 516, in read
    fileobj = ctx.__enter__()
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/utils/data.py", line 208, in get_readable_fileobj
    fileobj = io.FileIO(name_or_obj, 'r')
FileNotFoundError: [Errno 2] No such file or directory: '/global/homes/d/desi/desi_test_io/spectro/sim/test-quickgen/20150105/fibermap-00000100.fits'

======================================================================
ERROR: test_quickgen_moonphase_simspec (desisim.test.test_quickgen.TestQuickgen)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/test/test_quickgen.py", line 334, in test_quickgen_moonphase_simspec
    quickgen.main(quickgen.parse(cmd.split()[1:]))
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/scripts/quickgen.py", line 211, in main
    fibermap=read_fibermap(args.fibermap)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/code/desispec/master/py/desispec/io/fibermap.py", line 147, in read_fibermap
    return Table.read(filename, 'FIBERMAP')
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/table/table.py", line 2519, in read
    out = io_registry.read(cls, *args, **kwargs)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/io/registry.py", line 516, in read
    fileobj = ctx.__enter__()
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/utils/data.py", line 208, in get_readable_fileobj
    fileobj = io.FileIO(name_or_obj, 'r')
FileNotFoundError: [Errno 2] No such file or directory: '/global/homes/d/desi/desi_test_io/spectro/sim/test-quickgen/20150105/fibermap-00000100.fits'

======================================================================
ERROR: test_quickgen_moonzenith_simspec (desisim.test.test_quickgen.TestQuickgen)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/test/test_quickgen.py", line 390, in test_quickgen_moonzenith_simspec
    quickgen.main(quickgen.parse(cmd.split()[1:]))
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/scripts/quickgen.py", line 211, in main
    fibermap=read_fibermap(args.fibermap)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/code/desispec/master/py/desispec/io/fibermap.py", line 147, in read_fibermap
    return Table.read(filename, 'FIBERMAP')
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/table/table.py", line 2519, in read
    out = io_registry.read(cls, *args, **kwargs)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/io/registry.py", line 516, in read
    fileobj = ctx.__enter__()
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/utils/data.py", line 208, in get_readable_fileobj
    fileobj = io.FileIO(name_or_obj, 'r')
FileNotFoundError: [Errno 2] No such file or directory: '/global/homes/d/desi/desi_test_io/spectro/sim/test-quickgen/20150105/fibermap-00000100.fits'

======================================================================
ERROR: test_quickgen_seed_simspec (desisim.test.test_quickgen.TestQuickgen)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/test/test_quickgen.py", line 232, in test_quickgen_seed_simspec
    quickgen.main(quickgen.parse(cmd.split()[1:]))
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/scripts/quickgen.py", line 211, in main
    fibermap=read_fibermap(args.fibermap)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/code/desispec/master/py/desispec/io/fibermap.py", line 147, in read_fibermap
    return Table.read(filename, 'FIBERMAP')
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/table/table.py", line 2519, in read
    out = io_registry.read(cls, *args, **kwargs)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/io/registry.py", line 516, in read
    fileobj = ctx.__enter__()
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/global/common/software/desi/cori/desiconda/20180518-1.2.5-spec/conda/lib/python3.6/site-packages/astropy/utils/data.py", line 208, in get_readable_fileobj
    fileobj = io.FileIO(name_or_obj, 'r')
FileNotFoundError: [Errno 2] No such file or directory: '/global/homes/d/desi/desi_test_io/spectro/sim/test-quickgen/20150105/fibermap-00000100.fits'

======================================================================
FAIL: test_main_defaults (desisim.test.test_pixsim.TestPixsim)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/test/test_pixsim.py", line 203, in test_main_defaults
    self.assertTrue(os.path.exists(rawfile))
AssertionError: False is not true

======================================================================
FAIL: test_main_override (desisim.test.test_pixsim.TestPixsim)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/test/test_pixsim.py", line 244, in test_main_override
    self.assertTrue(os.path.exists(simpixfile))
AssertionError: False is not true

======================================================================
FAIL: test_parse (desisim.test.test_pixsim.TestPixsim)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/global/common/software/desi/cori/desiconda/20180130-1.2.4-spec/code/desisim/master/py/desisim/test/test_pixsim.py", line 277, in test_parse
    self.assertEqual(args.rawfile, desispec.io.findfile('raw', night, expid))
AssertionError: '/tmp/tmpmebvdh91/spectro/sim/test/20150105/00000124/desi-00000124.fits.fz' != '/tmp/tmpmebvdh91/spectro/sim/test/20150105/desi-00000124.fits.fz'
- /tmp/tmpmebvdh91/spectro/sim/test/20150105/00000124/desi-00000124.fits.fz
?                                            ---------
+ /tmp/tmpmebvdh91/spectro/sim/test/20150105/desi-00000124.fits.fz

[sbailey]

This turned out to be an error in desitest. desimodules/master is now using desiconda 20180518-1.2.5-spec for testing, which is ahead of the "current" link to the last stable release 20180130-1.2.4-spec. desitest was git updating the repos in the old area but the dependencies were coming from the new area which was not getting updated except by hand and were thus out of sync.

I updated desitest to git update the packages in whatever version of desiconda is loaded (determined via $DESICONDA), instead of updating the copies only in desiconda/current .