Issues running the example WASP14: pca problem

[AngelicaJara]

Hi! I installed Starfish in my laptop computer (Ubuntu 16.04). I followed the instructions and had no issues until trying to follow the example for WASP14.

The error appear when trying to run:
$ pca.py --create

$ File "pca.py", line 33, in
$ myHDF5 = HDF5Interface()
$ File "/home/angelica/Documentos/Starfish-master/Starfish/grid_tools.py", line 835, in init
$ self.grid_points = hdf5["pars"][:]
$ File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip-huypgcah-build/h5py/_objects.c:2840)
$ File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip-huypgcah-build/h5py/_objects.c:2798)
$ File "/home/angelica/.local/lib/python3.5/site-packages/h5py/_hl/group.py", line 169, in getitem
$ oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
$ File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/tmp/pip-huypgcah-build/h5py/_objects.c:2840)
$ File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/tmp/pip-huypgcah-build/h5py/_objects.c:2798)
$ File "h5py/h5o.pyx", line 190, in h5py.h5o.open (/tmp/pip-huypgcah-build/h5py/h5o.c:3734)
$ KeyError: "Unable to open object (Object 'pars' doesn't exist)"

Apparently the parameter 'pars' is not defined. I looked in the code 'grid_tools.py' and everything seems to be in order. I have not modified anything in the 'grid_tools.py' code.

I do not know what's wrong, I'd appreciate checking my problem.

[jason-neal]

What does your config.yaml file look like? I had issues #76 trying the tutorial last month because the default config.yaml had changed (so the tutorial did not specify everything that needed changing).

[AngelicaJara]

First I tried with the config file with two parameters and then with three parameters including metallicity, but with both the error was the same. In addition, the config file has the correct path.
I saw issues # 76 and the config file for the tutorial in the starfish demos. My config file is the same as the one you see there. So I do not know what the problem might be.
Thanks for your quick response. All suggestions are helpful.

[iancze]

Hi @AngelicaJara ,

I'm sorry that the configuration file is not working properly for you. Could you paste the contents of your config.yaml here or link to them via something like gist? It would be helpful for debugging your problem. Sometimes there may be formatting errors that creep in.

Given the h5py errors you are seeing, I am wondering if it may be an issue with specifying the correct path to the various intermediate grid products that are created.

Thank you.

[bersavosh]

I just tried running the example case, and while I did not encounter the specific issue mentioned here, I did run into other issues. Given the general name of this thread, I'm going to mention those here, instead of opening a new thread.

System configurations: Ubuntu 18.04.2, Python 3.7, and I installed Starfish from the git repo and using the same config file as suggested above.

1- pca.py contains multiple references to Starfish.parname, and it seem this no longer exists and based on looking at updates, I think this needs to be Starfish.config.parname. Though, I'm not entirely sure of this, thus I avoid doing a PR.

2- After I tweaked pca.py to reflect that, the commands in the example run smoothly until running pca.py --plot=emulator --params=fmin which leads to the following error:

Emulator parameters are:                                                                                                                                                                                           
lambda_xi 0.048577045459647084                                                                                                                                                                                     
[145.92283543 188.63358345   0.78408106   1.43236953]                                                                                                                                                              
[127.29339517 472.13142783   1.31384768   1.1860303 ]                                                                                                                                                              
[ 17.20499203 190.14242891   0.25104198   1.38437483]                                                                                                                                                              
[ 14.17852147 112.65298214   0.7459974    1.3598463 ]                                                                                                                                                              
[3.55077693e-06 1.39310526e+02 1.33731500e+00 1.33295662e+00]                                                                                                                                                      
multiprocessing.pool.RemoteTraceback:                                                                                                                                                                              
"""                                                                                                                                                                                                                
Traceback (most recent call last):                                                                                                                                                                                 
  File "/home/arash/anaconda3/envs/starfish/lib/python3.7/multiprocessing/pool.py", line 121, in worker                                                                                                            
    result = (True, func(*args, **kwds))                                                                                                                                                                           
  File "/home/arash/anaconda3/envs/starfish/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar                                                                                                            
    return list(map(*args))                                                                                                                                                                                        
  File "/home/arash/astro_sw/starfish/Starfish/scripts/pca.py", line 331, in plot_block                                                                                                                            
    active_dim = np.where(uni > 1)[0][0]                                                                                                                                                                           
IndexError: index 0 is out of bounds for axis 0 with size 0                                                                                                                                                        
"""                                                                                                                                                                                                                
                                                                                                                                                                                                                   
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/arash/astro_sw/starfish/Starfish/scripts/pca.py", line 381, in <module>
    p.map(plot_block, blocks)
  File "/home/arash/anaconda3/envs/starfish/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/arash/anaconda3/envs/starfish/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
IndexError: index 0 is out of bounds for axis 0 with size 0

Thanks,
Arash

[mileslucas]

@bersavosh

I've spent the last 8 months refactoring every part of Starfish, right now I am about 99% ready to merge my changes into master and push a release for v0.3.0. Would you try using that version of the code, since it will be the future for Starfish?

Here is the documentation: https://starfish.readthedocs.io/en/develop/
and the corresponding git branch is develop. To set up:

$ git clone https://github.com/iancze/starfish --branch=develop
$ cd starfish
$ pip install -e .

[bersavosh]

Thanks for the quick response, I will try this one and update.

[bersavosh]

@mileslucas

Thanks again for your response. I managed to install and run the example case for a single order spectrum. I did not find the file used in that example anywhere (hd109085_sxd_xcleanspec_120203.fits), so I grabbed another spectrum from the IRTF library and it works beautifully. This new version is a great update to an already great package. :-)

I think this issue can be closed when you releasev0.3.0, given that the issues discussed here are for older releases.

Just one last question: I don't see any documented way yet in v0.3.0 for fitting a continuum-normalized spectrum, is that possible (going to be possible)?

thanks!

[mileslucas]

Thank you for trying out the new version! Yes, that file is not in the repo because it is still proprietary to one of my collaborators and we need permission before hosting it online. I'm glad you got one of the archive stars to work, though.

As for continuum-normalization, we don't currently offer that but it is possible that we could add a keyword (eg continuum_normalize=True) to allow dividing out by a Planck curve. Unfortunately, since stars aren't true blackbodies, there's no guarantee that just dividing this Planck curve would actually normalize the spectrum to its continuum. Perhaps there are other, more fruitful ways of accomplishing this.

I personally haven't ever continuum normalized, though. @iancze can you weigh in?

[bersavosh]

In my experience, continuum normalization is useful to take care of problems like bad/non-existent flux calibration and some slit problems. But none of these issues would be addressed efficiently by a Planck curve normalization I believe. One could do polynomials with considering outliers in the wavelength range of interest (which is the refuge to seek when dealing with IRAF/PyRAF for these problems).

[iancze]

For the purposes of fitting a spectrum, continuum normalization is not necessary. The purpose of introducing the Chebyshev polynomials is to essentially fit for error in the flux calibration. Fitting continuum-normalized spectra directly without also allowing for small perturbations in the normalizing process (e.g., like with the Cheb polynomials) is tricky, since errors in the normalization process can bias your spectral fit.

That said, if you already have continuum-normalized spectra (i.e., you don't have access the raw or flux-calibrated spectra), you can still use Starfish. The flux normalization term (Omega, or solid angle) and the calibration polynomials will still adjust to bring the model closer to the data, it's just that the value of these parameters won't have any physical meaning.

[bersavosh]

I see, thanks!