-
Notifications
You must be signed in to change notification settings - Fork 115
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make Vaex the default HDF5 engine, add Exomol hdf5 support, add fetch_hitran() #398
Conversation
make files more up to date with Exojax parsers
…emp batch files & combining them) make vaex the default memory-mapping-engine for radis fix corresponding tests
This pull request introduces 3 alerts when merging 96125b7 into 01a6544 - view on LGTM.com new alerts:
|
fix multi-isotope vaex test
…used in Spyder IDE. Fixes radis#338 (the quick fix by changing `df.executor.async_method = "awaitio"` didn't work, and may have had side effects anyway)
- fetch_hitran & HITRANDatabaseManager implemented, use HAPI for download of all isotopes & all lines from HITRAN, and RADIS functions to parse & create local HDF5 formats - add HDF5 support in fetch_astroquery (which only fetches a partial range of HITRAN) - improved HDF5 support, fix guess_engine - add tests
…tch_databank : 'full' and 'range' default 'full'
This pull request introduces 5 alerts when merging 267ebdb into 01a6544 - view on LGTM.com new alerts:
|
This pull request introduces 5 alerts when merging 7978226 into 01a6544 - view on LGTM.com new alerts:
|
… databases) noneq require fetch_databank(load_columns='noneq') or load_databank(load_columns='noneq') calc_spectrum handles it alone
… new test-OH database with 'pytables' engine
Codecov Report
@@ Coverage Diff @@
## develop #398 +/- ##
===========================================
- Coverage 75.38% 75.12% -0.26%
===========================================
Files 156 157 +1
Lines 17673 18097 +424
===========================================
+ Hits 13323 13596 +273
- Misses 4350 4501 +151 |
This pull request introduces 4 alerts and fixes 2 when merging 9a7b598 into 01a6544 - view on LGTM.com new alerts:
fixed alerts:
|
# Conflicts: # environment.yml # radis/lbl/loader.py
This pull request introduces 4 alerts and fixes 2 when merging 93bbce7 into 92aae0d - view on LGTM.com new alerts:
fixed alerts:
|
This pull request introduces 4 alerts and fixes 2 when merging 1cc2d82 into 92aae0d - view on LGTM.com new alerts:
fixed alerts:
|
Description
MEMORY_MAPPING_ENGINE
(vaex
,pytables
) so users can change itAnd 2nd batch of changes
fetch_hitran()
which downloads all lines/isotopes of HITRAN database (Vaex compatible)Ex : Load full CO2 HITEMP database
Before (Pandas's Pytables) : ~ 85s
With Vaex : ~ 30s
With Vaex + load only required columns: ~4.5 s
So about ~8x faster than with all columns, 20x faster than initally, and of course reduces memory usage.