Fix tiles

[sbailey]

We have a new desi-tiles.fits file with a better dithering, but it broke code and tests for several reasons:

• new multi-dimensional columns that can't be exported to ecsv
• GRAY layer is not 0, not 4

This PR mostly fixes the code and tests to work with this new tile file, but I'm still having trouble with desimodel.test.test_footprint.test_partial_pixels, which sometimes passes and sometimes fails. The original test was already quite loose in testing the consistency between high and low resolution pixel weighting. @geordie666 could you take a look and help with that?

[dkirkby]

Have you tested the GRAY pass re-numbering with desisurvey and surveysim? This might not be the best moment for downtime on those packages while this is sorted out.

[moustakas]

I also wanted to ask about whether / how the footprint changed along the edges with this new tile file. Will we need all new mock catalogs or is the new footprint inclusive of the old one?

Apologies for jumping in with this question on this particular thread, but that could be another downstream wrinkle of this new tile file.

[sbailey]

desisurvey/surveysim: tests pass on my laptop but fail at NERSC due to a large negative MJD generated somewhere in the chain. The desisurvey rules files have not been updated to account for the new layer numbering. For the purposes of survey margin studies you could continue to use the latest desimodel tag.

Edge effects: the basic footprint is the same, but the new dithering has a much more scalloped pattern along the edges and I'm sure there are some healpix that come and go from new to old. We hadn't considered the impact on whether new mocks would need to be generated.

Worst case is that we back out to the old tile file as default while continuing to test the impact of the new one. Since we will need to do at least one more update to the footprint to match the final imaging, we should consider each of these cases for why re-defining the tiles isn't just a trivial replacement in the hopes that it will be easier in the next round.

[geordie666]

@sbailey: this unit test failure uncovered a (mild) bug. When passed a set of tiles, the desimodel.footprint.pixweight code was still defaulting to using the standard footprint to determine if a point was in DESI, rather than using the passed tiles. As the tiles being passed by the unit test were the old tiles, but the default tiles were now the new tiles, there was a discrepancy, which triggered the test failure.

I think I've fixed this, and, as a bonus, I made the test more realistic as the calculation is quicker once the pixweight code correctly handles the few tiles passed by the unit test, instead of defaulting to testing the entire footprint.

[sbailey]

Thanks @geordie666 we're in a state where merging this PR would fix desimodel to work with the new tiles, but we still have outstanding questions about the impact on desisurvey and mocks so I'm not going to merge just before going into the weekend.

[sbailey]

This PR is on hold while we get the mocks ready to use the new tiles. In the meantime, the new tiling file is in a desimodel svn branch "newtiles" and I've updated the travis tests of this PR to test against that instead of master.

[moustakas]

It's the case that this new tile file made its way into desimodel/master, right?

If so, this is causing issues with select_mock_targets since we haven't updated the mock catalogs to the new footprint. Can we make the master software release use a tag of the $DESIMODEL/data directory until this issue is resolved or some other solution?

Or do I need my own local checkout of $DESIMODEL?

source /project/projectdirs/desi/software/desi_environment.sh master
cd $DESIMODEL/data
svn info
Path: .
Working Copy Root Path: /global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/code/desimodel/master/data
URL: https://desi.lbl.gov/svn/code/desimodel/trunk/data
Relative URL: ^/desimodel/trunk/data
Repository Root: https://desi.lbl.gov/svn/code
Repository UUID: 10003831-d790-44f6-a3bc-8e9ec13a4ac3
Revision: 119772

[sbailey]

@moustakas something else is wrong. I restored the previous tile file into desimodel svn trunk last week, and on the code side PR #95 hasn't been merged, so master should be ok with the old tile file. What is the symptom that made you think we still had the new tile file in master/trunk?

[cori11 desimodel] pwd
/global/common/software/desi/cori/desiconda/20180709-1.2.6-spec/code/desimodel
[cori11 desimodel] fitsdiff 0.9.9/data/footprint/desi-tiles.fits master/data/footprint/desi-tiles.fits

 fitsdiff: 2.0.7
 a: 0.9.9/data/footprint/desi-tiles.fits
 b: master/data/footprint/desi-tiles.fits
 Maximum number of different data values to be reported: 10
 Relative tolerance: 0.0, Absolute tolerance: 0.0

No differences found.

[weaverba137]

@sbailey, is this ever going to be merged? Also, is there any reason to keep the already-merged branch gfa?

[sbailey]

Some version of this branch will be needed to support the data model changes of the new tile file in the SVN newtiles branch. I do want to do that, but I'm trying to get everything else back into a working state and then introduce the new tiles as a single isolated change instead of throwing it into the broken mix too.

I just deleted the state gfa branch.

[weaverba137]

Thank you. Sounds like I can ignore this PR for the purposes of a 19.9 release then.

[moustakas]

Can we resurrect this issue? In particular, if this new tile file has knowledge of the final imaging footprint (as defined by DR8) that would be very useful for DR9 prep. @sbailey @djschlegel

[weaverba137]

Could someone please summarize the remaining issues that need to be addressed before this can be merged? This branch is sufficiently old that it may need to be rebased on master (that's something I can do).

[sbailey]

closing this stale PR. Will open a new PR that supersedes this.

[weaverba137]

Do we need to keep the fix-tiles branch? It looks like #135 replaced this PR.