-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bias correction #292
Bias correction #292
Changes from 15 commits
707b263
e817f7b
0c57bb4
b5a3566
a125ff4
26d62d1
a17dc6d
d0849e7
3f09f0a
4fbb8df
687e203
f187dde
e129a39
66b67e3
dc53a33
73b2949
32cd217
a08cc05
11b7249
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,5 @@ | ||
language: generic | ||
|
||
os: linux | ||
dist: xenial | ||
|
||
jobs: | ||
|
@@ -15,8 +15,7 @@ jobs: | |
update: true | ||
env: | ||
- CONDA_FN="Miniconda3-latest-MacOSX-x86_64.sh" | ||
- os: linux | ||
name: "Linux (Python3.6 + pep8)" | ||
- name: "Linux (Python3.6 + pep8)" | ||
dist: xenial | ||
python: "3.6" | ||
addons: | ||
|
@@ -29,18 +28,15 @@ jobs: | |
- os: linux | ||
name: "Linux (Python3.7 + docs)" | ||
python: "3.7" | ||
dist: xenial | ||
addons: | ||
apt: | ||
packages: | ||
- libnetcdf-dev | ||
env: | ||
- CONDA_FN="Miniconda3-latest-Linux-x86_64.sh" | ||
- DOCS=true | ||
- os: linux | ||
name: "Linux (Python3.8 + notebooks)" | ||
- name: "Linux (Python3.8 + notebooks)" | ||
python: "3.8" | ||
dist: xenial | ||
addons: | ||
apt: | ||
packages: | ||
|
@@ -56,6 +52,7 @@ jobs: | |
branches: | ||
only: | ||
- master | ||
- bias_correction | ||
|
||
install: | ||
- make bootstrap | ||
|
@@ -75,8 +72,9 @@ install: | |
|
||
before_script: | ||
# # Start WPS service on port 5000 on 0.0.0.0 | ||
# - raven start --daemon --bind-host 0.0.0.0 --port 5000 | ||
- bash -c "source $HOME/miniconda3/bin/activate raven && make start" | ||
# - raven start --daemon --bind-host 0.0.0.0 --port 5000 | ||
- bash -c "source $HOME/miniconda3/bin/activate raven && pip install git+https://github.com/pydata/xarray.git@master" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this install from xarray.git@master still necessary with xarray 0.16 from There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. With xarray at 0.16, the sdba functions that Pascal was working on are integrated, so no. This workaround can blissfully be thrown in the trash. |
||
- bash -c "source $HOME/miniconda3/bin/activate raven && make start" | ||
|
||
script: | ||
- bash -c "source $HOME/miniconda3/bin/activate raven && make test" | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,34 @@ | ||
import datetime as dt | ||
import xarray as xr | ||
import xclim.sdba as sdba | ||
|
||
|
||
class TestBiasCorrect: | ||
def test_bias_correction(self): | ||
|
||
ref_data='https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/datasets/simulations/cmip5/atmos/day_MPI-ESM-LR_historical.ncml' | ||
fut_data='https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/datasets/simulations/cmip5/atmos/day_MPI-ESM-LR_historical+rcp85.ncml' | ||
hist_data="https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/datasets/gridded_obs/nrcan_v2.ncml" | ||
|
||
lat=54.484 | ||
lon=-123.36 | ||
|
||
# CAREFUL! ERA5 IS NOT ThE SAME LONGITUDE | ||
# Also, latitude goes from high to low, so I need to have it go from lat+1 to lat-1 in the slice. | ||
# For the NRCan dataset, I cannot have more than about 10 years as I get a "NetCDF: DAP failure" which I think is related to a timeout. | ||
ds = (xr.open_dataset(hist_data).sel(lat=slice(lat + 1, lat - 1),lon=slice(lon - 1, lon + 1), time=slice(dt.datetime(1991,1,1), dt.datetime(2010,12,31))).mean(dim={"lat", "lon"}, keep_attrs=True)) | ||
|
||
# For lon in 0-360 format, need to add an auto-checker. | ||
lon = 260 | ||
ds2 = (xr.open_dataset(ref_data).sel(lat=slice(lat - 1, lat + 1),lon=slice(lon - 1, lon + 1), time=slice(dt.datetime(1981,1,1), dt.datetime(2010,12,31))).mean(dim={"lat", "lon"}, keep_attrs=True)) | ||
ds3 = (xr.open_dataset(fut_data).sel(lat=slice(lat - 1, lat + 1),lon=slice(lon - 1, lon + 1), time=slice(dt.datetime(2041,1,1), dt.datetime(2070,12,31))).mean(dim={"lat", "lon"}, keep_attrs=True)) | ||
|
||
# Here data in ds, ds2 and ds3 are NaN! | ||
|
||
group_month_nowindow = sdba.utils.Grouper('time.month') | ||
Adj = sdba.DetrendedQuantileMapping(nquantiles=50, kind='+', group=group_month_nowindow) | ||
Adj.train(ds['pr'],ds2['pr']) | ||
Scen = Adj.adjust(ds3['pr'], interp="linear") | ||
Adj.ds.af # adjustment factors. | ||
|
||
print(Adj.ds.af) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we want to limit Travis-CI to only
master
branch, then we have to add each dev branch manually, like thisbias_correction
branch? @Zeitsperre (b5a3566)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This can be removed. It was placed in order to trigger the bias_correction branch that PRs were merging to in a few instances. The reason why I set only to master is that otherwise we tend to get two duplicate sets of builds; one that is the pushes set against master and another that is the pushes set against the last version of master it stemmed from.
Since we only care about the current state of master, as all pushes must integrate these changes, there's nothing to gain except more redundant build checks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the redundant build checks on
master
branch will happen only once, when the PR is merged tomaster
.Whereas instructions to build only the
master
branch will not test any commits on any devel branches and force all devel branches to hardcode the devel branch name like we just saw.I think having the redundant build on
master
is a less bad of the 2 problems. Wonder what's your opinion about removing this "build onlymaster
branch" instruction for all birds?