Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Vaex the default HDF5 engine, add Exomol hdf5 support, add fetch_hitran() #398

Merged
merged 18 commits into from Nov 14, 2021

Conversation

erwanp
Copy link
Member

@erwanp erwanp commented Nov 13, 2021

Description

And 2nd batch of changes

  • ⭐ add fetch_hitran() which downloads all lines/isotopes of HITRAN database (Vaex compatible)
from radis.io.hitran import fetch_hitran
df = fetch_hitran("CO")
  • eventually, keep the choice between 2 HITRAN retrieval modes: 'full' (new, new default) and 'range' (old, download only required range)
calc_spectrum(..., databank=("hitran", "full"))   # predownload full database & isotopes (once for all)
calc_spectrum(..., databank=("hitran", "range"))   # download only the required the range & isotope
  • ⭐ only load required columns by default (makes it 8-10x faster to load databases)

Ex : Load full CO2 HITEMP database

from radis import calc_spectrum
s = calc_spectrum(1, 26777,         # cm-1           # <<< full spectral range!
                  molecule='CO2',
                  isotope='1,2,3',
                  pressure=760 * u.torr,
                  Tgas=700,           # K
                  mole_fraction=1,
                  wstep=0.01,
                  databank='hitemp',
                  verbose=3,
                  cutoff=0,

                  )
s.print_perf_profile()
s.plot(yscale='log')

Before (Pandas's Pytables) : ~ 85s
With Vaex : ~ 30s
With Vaex + load only required columns: ~4.5 s

So about ~8x faster than with all columns, 20x faster than initally, and of course reduces memory usage.

@erwanp erwanp added performance db:exomol related to the ExoMol database labels Nov 13, 2021
@erwanp erwanp added this to the 0.11.0 milestone Nov 13, 2021
@lgtm-com
Copy link

lgtm-com bot commented Nov 13, 2021

This pull request introduces 3 alerts when merging 96125b7 into 01a6544 - view on LGTM.com

new alerts:

  • 2 for Except block handles 'BaseException'
  • 1 for Wrong number of arguments in a call

…used in Spyder IDE. Fixes radis#338

(the quick fix by changing `df.executor.async_method = "awaitio"` didn't work, and may have had side effects anyway)
@erwanp erwanp mentioned this pull request Nov 14, 2021
84 tasks
- fetch_hitran & HITRANDatabaseManager implemented, use HAPI for download of all isotopes & all lines from HITRAN, and RADIS functions to parse & create local HDF5 formats
- add HDF5 support in fetch_astroquery (which only fetches a partial range of HITRAN)
- improved HDF5 support, fix guess_engine
- add tests
…tch_databank : 'full' and 'range'

default 'full'
changes in docstrings
@lgtm-com
Copy link

lgtm-com bot commented Nov 14, 2021

This pull request introduces 5 alerts when merging 267ebdb into 01a6544 - view on LGTM.com

new alerts:

  • 2 for Except block handles 'BaseException'
  • 2 for Module is imported with 'import' and 'import from'
  • 1 for Wrong number of arguments in a call

@lgtm-com
Copy link

lgtm-com bot commented Nov 14, 2021

This pull request introduces 5 alerts when merging 7978226 into 01a6544 - view on LGTM.com

new alerts:

  • 2 for Except block handles 'BaseException'
  • 2 for Module is imported with 'import' and 'import from'
  • 1 for Wrong number of arguments in a call

… databases)

noneq require fetch_databank(load_columns='noneq') or load_databank(load_columns='noneq')
calc_spectrum handles it alone
… new test-OH database with 'pytables' engine
@codecov-commenter
Copy link

Codecov Report

Merging #398 (9a7b598) into develop (eb9258b) will decrease coverage by 0.25%.
The diff coverage is 67.93%.

@@             Coverage Diff             @@
##           develop     #398      +/-   ##
===========================================
- Coverage    75.38%   75.12%   -0.26%     
===========================================
  Files          156      157       +1     
  Lines        17673    18097     +424     
===========================================
+ Hits         13323    13596     +273     
- Misses        4350     4501     +151     

@lgtm-com
Copy link

lgtm-com bot commented Nov 14, 2021

This pull request introduces 4 alerts and fixes 2 when merging 9a7b598 into 01a6544 - view on LGTM.com

new alerts:

  • 2 for Except block handles 'BaseException'
  • 1 for Module is imported with 'import' and 'import from'
  • 1 for Wrong number of arguments in a call

fixed alerts:

  • 1 for Module is imported more than once
  • 1 for Missing named arguments in formatting call

# Conflicts:
#	environment.yml
#	radis/lbl/loader.py
@lgtm-com
Copy link

lgtm-com bot commented Nov 14, 2021

This pull request introduces 4 alerts and fixes 2 when merging 93bbce7 into 92aae0d - view on LGTM.com

new alerts:

  • 2 for Except block handles 'BaseException'
  • 1 for Module is imported with 'import' and 'import from'
  • 1 for Wrong number of arguments in a call

fixed alerts:

  • 1 for Module is imported more than once
  • 1 for Missing named arguments in formatting call

@erwanp erwanp merged commit f9c433c into radis:develop Nov 14, 2021
@erwanp erwanp mentioned this pull request Nov 14, 2021
@lgtm-com
Copy link

lgtm-com bot commented Nov 14, 2021

This pull request introduces 4 alerts and fixes 2 when merging 1cc2d82 into 92aae0d - view on LGTM.com

new alerts:

  • 2 for Except block handles 'BaseException'
  • 1 for Module is imported with 'import' and 'import from'
  • 1 for Wrong number of arguments in a call

fixed alerts:

  • 1 for Module is imported more than once
  • 1 for Missing named arguments in formatting call

@erwanp erwanp changed the title Make Vaex the default HDF5 engine, add Exomol hdf5 support Make Vaex the default HDF5 engine, add Exomol hdf5 support, add fetch_hitran() Nov 14, 2021
@erwanp erwanp deleted the add/exomol-hdf5-support branch November 16, 2021 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
db:exomol related to the ExoMol database performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Broadening of ExoMol molecules config file with comments
2 participants