Add .splups file reader

[jwreep]

Fixes #17

(There are a couple of notes in that thread about how to modify functions to use .splups files that this does not address. I've only added the file reader here.)

Tested with data from version 7.1. I tried both C V and C VI since their .splups files have different lengths.

>>> from astropy.table import QTable
>>> import fiasco.io
>>> parser=fiasco.io.Parser('/Users/reep/ssw/packages/chianti/dbase_7.1/c/c_5/c_5.splups')
>>> table = parser.parse()
>>> table
<QTable length=139>
  Z    ion  lower_level ... delta_energy   bt_c        bt_rate       
                        ...      Ry                                  
int64 int64    int64    ...   float64    float64      object[5]      
----- ----- ----------- ... ------------ ------- --------------------
    6     5           1 ...        21.77     0.7  0.007704 .. 0.01205
    6     5           1 ...        22.29     1.7   0.01627 .. 0.02856
    6     5           1 ...        22.22     0.7 0.004817 .. 0.005243
    6     5           1 ...        22.22     1.7   0.01434 .. 0.01721
    6     5           1 ...        22.23     1.7   0.03345 .. 0.04014
    6     5           1 ...        22.63     2.5     0.03161 .. 0.112
  ...   ...         ... ...          ...     ...                  ...
    6     5           2 ...       0.3999     3.5       4.695 .. 1.346
    6     5           2 ...       0.4012     3.5        7.815 .. 2.26
    6     5           3 ...       0.2583     3.5       6.371 .. 1.426
    6     5           4 ...       0.2571    14.4  0.1364 .. 0.0005553
    6     5           5 ...       0.2572    14.2    0.4345 .. 0.02541
    6     5           6 ...       0.2559    14.2     0.7983 .. 0.0398
>>> table.meta
OrderedDict([('footer', "filename: c_5.splups\noscillator strengths and A values (transitions from upper configurations 1s.2p,1s.3p,1s.3d):\nMartin,W.C., Sugar,J., Musgrove,A., Dalton,G.R., 1995,\nNIST Database for Atomic Spectroscopy, Version 1.0, NIST Standard Reference Database 61\noscillator strengths (transitions from 1s.3s, 1snl with n>3): Wiese, W.L., Smith, M.W., Glennon, B.M., 1966,\nAtomic Transition Probabilities, NSRDS-NBS-4\ncomment:  Wiese's oscillator strengths are hydrogenic\ncollision strengths: Sampson, D.H., Goett, S.J., Clark, R.E.H., 1983, ADNDT, 29, 467 (Hydrogenic values)\ncollision strengths among n=2 levels: extrapolated from Zhang and Sampson (1987, ApJSS, 63, 487) values for\nO VII, Ne IX, Mg XI\ncomment: correction (Aug 2003): a few more collisional transitions have been added.\nproduced as part of the Arcetri/Cambridge/NRL 'CHIANTI' atomic data base collaboration\nK.P. Dere  - August 2003"), ('chianti_version', ''), ('filename', '/Users/reep/ssw/packages/chianti/dbase_7.1/c/c_5/c_5.splups'), ('descriptions', {'Z': 'atomic number', 'ion': 'ionization state', 'lower_level': 'lower level index', 'upper_level': 'upper level index', 'bt_type': 'Burgess-Tully scaling type', 'gf': 'oscillator strength', 'delta_energy': 'delta energy', 'bt_c': 'Burgess-Tully scaling parameter', 'bt_rate': 'Burgess-Tully scaled effective collision strengths'}), ('element', 'c'), ('ion', 'c_5'), ('dielectronic', False)])
>>> parser=fiasco.io.Parser('/Users/reep/ssw/packages/chianti/dbase_7.1/c/c_6/c_6.splups')
>>> table = parser.parse()
>>> table
<QTable length=48>
  Z    ion  lower_level ... delta_energy   bt_c         bt_rate       
                        ...      Ry                                   
int64 int64    int64    ...   float64    float64       object[9]      
----- ----- ----------- ... ------------ ------- ---------------------
    6     6           1 ...         27.0   0.169    0.01761 .. 0.02252
    6     6           1 ...         27.0    1.43    0.02356 .. 0.04101
    6     6           1 ...         27.0    1.43      0.04712 .. 0.082
    6     6           1 ...         32.0     0.1    0.0055 .. 0.004458
    6     6           1 ...         32.0    1.43    0.005339 .. 0.0066
    6     6           1 ...         32.0    1.43     0.01067 .. 0.0132
  ...   ...         ... ...          ...     ...                   ...
    6     6           2 ...         7.56  0.4457    0.02488 .. 0.01625
    6     6           2 ...         7.56  0.2255   0.00973 .. 0.001067
    6     6           2 ...         7.56    0.21   0.01218 .. 0.001335
    6     6           3 ...    0.0002384    98.0 0.005185 .. 0.0005433
    6     6           2 ...     0.004089  1492.0 0.006689 .. 0.0008394
    6     6           3 ...     0.004328  1412.0  0.000221 .. 8.27e-06
>>> table.meta
OrderedDict([('footer', "filename: c_6.splups\noscillator strengths: SUPERSTRUCTURE calculation including all levels up to n=5.\neffective collision strengths: Ballance, Badnell, & Smith 2003, J. Phys. B, 36, 3707\ncomment: fine-structure values for electron impact-excitation within the n=2 levels from Zygelman & Dalgarno (1987) Phys. Rev. A, 35, 4085\ncomment: The fine structure collision strengths have been obtained assuming a distribution according\nto statistical weights.\ncomment: Only excitations from the 1s and 2s levels are retained from the original dataset.\nproduced as part of the Arcetri/Cambridge/NRL 'CHIANTI' atomic data base collaboration\nG. Del Zanna  -  Nov  2007"), ('chianti_version', ''), ('filename', '/Users/reep/ssw/packages/chianti/dbase_7.1/c/c_6/c_6.splups'), ('descriptions', {'Z': 'atomic number', 'ion': 'ionization state', 'lower_level': 'lower level index', 'upper_level': 'upper level index', 'bt_type': 'Burgess-Tully scaling type', 'gf': 'oscillator strength', 'delta_energy': 'delta energy', 'bt_c': 'Burgess-Tully scaling parameter', 'bt_rate': 'Burgess-Tully scaled effective collision strengths'}), ('element', 'c'), ('ion', 'c_6'), ('dielectronic', False)])

[codecov]

Codecov Report

Attention: Patch coverage is 79.54545% with 9 lines in your changes are missing coverage. Please review.

Project coverage is 92.43%. Comparing base (73e3228) to head (eee3efd).

Files	Patch %	Lines
fiasco/io/sources/ion_sources.py	50.00%	7 Missing ⚠️
fiasco/conftest.py	85.71%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #278      +/-   ##
==========================================
- Coverage   93.08%   92.43%   -0.65%     
==========================================
  Files          38       38              
  Lines        2863     2896      +33     
==========================================
+ Hits         2665     2677      +12     
- Misses        198      219      +21     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jwreep]

I'm guessing CodeCov's issue is because the tests are for v8.0.6 and this requires < 8 to test?

[wtbarnes]

I believe that's correct. There is code being added, but the tests that cover those lines are not being run on the CI.

Out of curiosity, do the tests pass if you run them against the v7 database? i.e.

pytest fiasco --ascii-dbase-root ~/.chianti --ascii-dbase-url http://download.chiantidatabase.org/CHIANTI_v7.1.4_database.tar.gz --disable-file-hash --skip-version-check

If they do, we should add an additional test to the CI for v7.

[jwreep]

They do not pass, but it looks like it's because all of the .scups files are missing:

=========================== short test summary info ============================
FAILED fiasco/io/sources/tests/test_sources.py::test_ion_sources[h_1.scups] - fiasco.util.exceptions.MissingASCIIFileError: Could not find file /Users/re...
FAILED fiasco/tests/test_collections.py::test_two_photon[wavelength0] - IndexError: index 0 is out of bounds for axis 0 with size 0
FAILED fiasco/tests/test_collections.py::test_two_photon[wavelength1] - IndexError: index 0 is out of bounds for axis 0 with size 0
FAILED fiasco/tests/test_collections.py::test_radiative_loss - assert False
FAILED fiasco/tests/test_collections.py::test_spectrum - ValueError: No collision or transition data available for any ion in collec...
FAILED fiasco/tests/test_ion.py::test_level - assert <Quantity 0> == 5
FAILED fiasco/tests/test_ion.py::test_proton_collision - assert False
FAILED fiasco/tests/test_ion.py::test_level_populations - fiasco.util.exceptions.MissingDatasetException: _scups dataset missing for ...
FAILED fiasco/tests/test_ion.py::test_level_populations_proton_data_toggle - fiasco.util.exceptions.MissingDatasetException: _scups dataset missing for ...
FAILED fiasco/tests/test_ion.py::test_contribution_function - fiasco.util.exceptions.MissingDatasetException: _scups dataset missing for ...
FAILED fiasco/tests/test_ion.py::test_emissivity_shape[density0-False] - fiasco.util.exceptions.MissingDatasetException: _scups dataset missing for ...
FAILED fiasco/tests/test_ion.py::test_emissivity_shape[density1-False] - fiasco.util.exceptions.MissingDatasetException: _scups dataset missing for ...
FAILED fiasco/tests/test_ion.py::test_emissivity_shape[None-True] - fiasco.util.exceptions.MissingDatasetException: _scups dataset missing for ...
FAILED fiasco/tests/test_ion.py::test_coupling_unequal_dimensions_exception - fiasco.util.exceptions.MissingDatasetException: _scups dataset missing for ...
FAILED fiasco/tests/test_ion.py::test_emissivity - fiasco.util.exceptions.MissingDatasetException: _scups dataset missing for ...
FAILED fiasco/tests/test_ion.py::test_intensity[em0] - fiasco.util.exceptions.MissingDatasetException: _scups dataset missing for ...
FAILED fiasco/tests/test_ion.py::test_intensity[em1] - fiasco.util.exceptions.MissingDatasetException: _scups dataset missing for ...
FAILED fiasco/tests/test_ion.py::test_intensity[em2] - fiasco.util.exceptions.MissingDatasetException: _scups dataset missing for ...
FAILED fiasco/tests/test_ion.py::test_two_photon - IndexError: index 0 is out of bounds for axis 0 with size 0
FAILED fiasco/tests/test_ion.py::test_transitions - assert False
ERROR fiasco/tests/test_ion.py::test_level_populations_normalized - fiasco.util.exceptions.MissingDatasetException: _scups dataset missing for ...
ERROR fiasco/tests/test_ion.py::test_level_populations_correction - fiasco.util.exceptions.MissingDatasetException: _scups dataset missing for ...
============ 20 failed, 130 passed, 20 skipped, 2 errors in 18.57s =============

I suppose to test the older versions, these functions would need to be ignored.

[wtbarnes]

Ok that's not too surprising. We could just use the requires_dbase_version mark to skip these if we're running on a version less than 8. Once we actually support using the information in the splups files to do the needed calculation, we could unskip anything that doesn't specifically test reading the scups files.

[jwreep]

Right, I've updated the tests to ignore the ones that require .scups files for now. I had to modify the fixture to allow multiple conditions (two functions require version > 7 and < 9). I've added a test for version 7 of the database.

The version 9 test is failing now, though, so perhaps I've done the fixture incorrectly? I can't see the root of the issue right now. It runs correctly in the command line...

[wtbarnes]

Ok I think I've fixed it. There was a small bug in handling multiple conditions, but I went ahead and just refactored that mark as my original implementation was clumsy anyway. This also makes using the marks a bit less awkward, i.e. you just specify a string rather than always needing two string which makes handling multiple conditions easier.

This does implicitly assume the conditions are joined by an AND as opposed to an OR, but I think that's fine for now.

[jwreep]

Thanks for the help! My coding was probably a bit awkward there.

[wtbarnes]

No worries. My initial implementation made it awkward to extend to multiple conditions.

[wtbarnes]

Thanks as always for taking this on!