Add ability to insist on using IDL-refined AHF catalogues

[apontzen]

Add ability to insist on using IDL-refined AHF catalogues (Alyson Brooks' toolset),
or to insist on ignoring those catalogues. This is in response to issue #117, where
it has become clear people have both types of catalogue on disk and this can cause
severe confusion due to the automated search for catalogues and statistics files.

To insist on using IDL-generated catalogues, use

tangos add <name_of_sim> --handler=pynbody.ChangaUseIDLInputHandler

To insist on ignoring IDL-generated catalogues, use

tangos add <name_of_sim> --handler=pynbody.ChangaIgnoreIDLInputHandler

The default behaviour remains unchanged (it will guess the best files to use,
which can lead to confusion if there is more than one type of catalogue
hanging around on disk)

[mtremmel]

@apontzen quick question. Will the handler have to be specified for all subsequent calculations too?

also, can you ping me when this is resolved and I can start working on the AHF branch again to finalize that? Thanks!

[apontzen]

No no, the handler is associated with the simulation when you add it. It's automatically used after that. @trquinn could you confirm whether this helps for your situation? Then I can merge.

[apontzen]

Pinging @trquinn to see whether we can merge this. In the meantime @mtremmel you can always develop by branching off here, instead of off the master branch.

[mtremmel]

I already have most of the code done on a branch tracking master so much easier to wait. Still, I'd rather play it safe and wait until this is done so I don't end up screwing anything up :-)

[trquinn]

starting a run now (it will take a day or two to complete).

[apontzen]

Thanks @trquinn -- just to alert you, though, that if you are using this branch on a production run, it does not currently have the fix for #121 (it is based off an earlier version of the master branch).

[trquinn]

With tangos add, I get:

  File "/home/u14266/.local/lib/python2.7/site-packages/tangos-1.0.10-py2.7.egg/
tangos/scripts/__init__.py", line 49, in main
    args.func(args)
  File "/home/u14266/.local/lib/python2.7/site-packages/tangos-1.0.10-py2.7.egg/
tangos/scripts/manager.py", line 34, in add_simulation_timesteps
    adder.scan_simulation_and_add_all_descendants()
  File "/home/u14266/.local/lib/python2.7/site-packages/tangos-1.0.10-py2.7.egg/
tangos/tools/add_simulation.py", line 33, in scan_simulation_and_add_all_descend
ants
    self.add_simulation_properties()
  File "/home/u14266/.local/lib/python2.7/site-packages/tangos-1.0.10-py2.7.egg/
tangos/tools/add_simulation.py", line 67, in add_simulation_properties
    properties_dict = self.simulation_output.get_properties()
  File "/home/u14266/.local/lib/python2.7/site-packages/tangos-1.0.10-py2.7.egg/
tangos/input_handlers/pynbody.py", line 478, in get_properties
    parent_prop_dict = super(ChangaInputHandler, self).get_properties()
  File "/home/u14266/.local/lib/python2.7/site-packages/tangos-1.0.10-py2.7.egg/
tangos/input_handlers/pynbody.py", line 258, in get_properties
    timesteps = list(self.enumerate_timestep_extensions())
  File "/home/u14266/.local/lib/python2.7/site-packages/tangos-1.0.10-py2.7.egg/
tangos/input_handlers/finding.py", line 67, in enumerate_timestep_extensions
    if self._is_able_to_load(e):
  File "/home/u14266/.local/lib/python2.7/site-packages/tangos-1.0.10-py2.7.egg/tangos/input_handlers/pynbody.py", line 68, in _is_able_to_load
    h = self._construct_pynbody_halos(f)
  File "/home/u14266/.local/lib/python2.7/site-packages/tangos-1.0.10-py2.7.egg/tangos/input_handlers/pynbody.py", line 59, in _construct_pynbody_halos
    return halo_class(sim, *args, **kwargs)
  File "/home/u14266/.local/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/halo/__init__.py", line 271, in __init__
    GrpCatalogue.__init__(self, sim, arr_name)
  File "/home/u14266/.local/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/halo/__init__.py", line 181, in __init__
    sim[array] # trigger lazy-loading and/or kick up a fuss if unavailable
  File "/home/u14266/.local/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py", line 263, in __getitem__
    return self._get_array_with_lazy_actions(i)
  File "/home/u14266/.local/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py", line 358, in _get_array_with_lazy_actions
    return self._get_array(name)
  File "/home/u14266/.local/lib/python2.7/site-packages/pynbody-0.47-py2.7-linux-x86_64.egg/pynbody/snapshot/__init__.py", line 1264, in _get_array
    x = self._arrays[name]
KeyError: 'amiga.grp'

[apontzen]

That looks correct (sort of) -- you are asking it to use the .amiga.grp files, and it is attempting to do so. But it can't find one of them on disk. Are they definitely all there?

[apontzen]

@mtremmel @trquinn Any update on using this in a real-world setting? Not sure whether to merge it or not.

[trquinn]

I'm having trouble testing this since, for some reason, I don't have all the amig.grp files. It looks like either AHF or the IDL failed with some timesteps. I did test the --handler=pynbody.ChangaIgnoreIDLInputHandler flag, and that seems to work.

[mtremmel]

Hey just a ping on this... should it be merged soon?

[apontzen]

I merged this rather than let it hang any longer. Even though we didn't seem to fully test it in the real world I believe it should work and we can reopen an issue if not.