Load data from transient and scenario run into DataFrame

[tobeycarman]

Assuming you have already run the model and you have a folder of outputs, this function will load the data into a pandas.DataFrame.

The example here works on the development docker container.

develop@809ec5355e8e: cd /data/workflows/TEM-RS/Piksiksak_iem_veg_map/output
develop@809ec5355e8e: /data/workflows/TEM-RS/Piksiksak_iem_veg_map/output$ ls
ALD_yearly_sc.nc     DEEPC_yearly_sp.nc   GPP_monthly_tr.nc        LAYERTYPE_yearly_sc.nc  RG_monthly_sp.nc  RM_monthly_tr.nc     TLAYER_monthly_sc.nc  restart-pr.nc
ALD_yearly_sp.nc     DEEPC_yearly_tr.nc   LAYERDEPTH_yearly_sc.nc  LAYERTYPE_yearly_sp.nc  RG_monthly_tr.nc  SHLWC_yearly_sc.nc   TLAYER_monthly_sp.nc  restart-sc.nc
ALD_yearly_tr.nc     DEEPDZ_yearly_sc.nc  LAYERDEPTH_yearly_sp.nc  LAYERTYPE_yearly_tr.nc  RH_monthly_sc.nc  SHLWC_yearly_sp.nc   TLAYER_monthly_tr.nc  restart-sp.nc
CMTNUM_yearly_sc.nc  DEEPDZ_yearly_sp.nc  LAYERDEPTH_yearly_tr.nc  MINEC_yearly_sc.nc      RH_monthly_sp.nc  SHLWC_yearly_tr.nc   VEGC_yearly_sc.nc     restart-tr.nc
CMTNUM_yearly_sp.nc  DEEPDZ_yearly_tr.nc  LAYERDZ_yearly_sc.nc     MINEC_yearly_sp.nc      RH_monthly_tr.nc  SHLWDZ_yearly_sc.nc  VEGC_yearly_sp.nc     run_status.nc
CMTNUM_yearly_tr.nc  GPP_monthly_sc.nc    LAYERDZ_yearly_sp.nc     MINEC_yearly_tr.nc      RM_monthly_sc.nc  SHLWDZ_yearly_sp.nc  VEGC_yearly_tr.nc
DEEPC_yearly_sc.nc   GPP_monthly_sp.nc    LAYERDZ_yearly_tr.nc     RG_monthly_sc.nc        RM_monthly_sp.nc  SHLWDZ_yearly_tr.nc  restart-eq.nc

Now start IPython and load the utility script:

develop@809ec5355e8e:/work$ ipython
Python 3.8.6 (default, Jul 17 2023, 21:04:01) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.10.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import scripts.output_utils as ou

Then load the data:

In [8]: df, meta = ou.load_trsc_dataframe('GPP', 'monthly', 0,0, '/data/workflows/TEM-RS/Piksiksak_iem_veg_map/output/')
Trying to open:  /data/workflows/TEM-RS/Piksiksak_iem_veg_map/output/GPP_monthly_tr.nc
Trying to open:  /data/workflows/TEM-RS/Piksiksak_iem_veg_map/output/GPP_monthly_sc.nc

And take a look at it:

In [9]: df.head()
Out[9]: 
              0    1    2    3    4    5    6    7    8    9
1901-01-01  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
1901-02-01  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
1901-03-01  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
1901-04-01  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0
1901-05-01  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0  0.0

In [10]: df.tail()
Out[10]: 
                    0         1          2          3         4         5          6          7    8    9
2100-08-01  28.642951  0.658936  61.360704  89.711769  0.551334  5.101302  17.399002  12.236366  0.0  0.0
2100-09-01  17.893332  0.426614  38.662317  57.530792  0.359265  3.703657  12.464457   8.777431  0.0  0.0
2100-10-01   1.194852  0.034975   2.659676   4.055428  0.029883  0.702443   2.307354   1.631811  0.0  0.0
2100-11-01   0.000000  0.000000   0.000000   0.000000  0.000000  0.000000   0.000000   0.000000  0.0  0.0
2100-12-01   0.000000  0.000000   0.000000   0.000000  0.000000  0.000000   0.000000   0.000000  0.0  0.0

In [11]: meta
Out[11]: {'var_units': 'g/m2/month'}