python
import numpy as np import pandas as pd import xarray as xr
np.random.seed(123456)
- For combining datasets or data arrays along a single dimension, see concatenate.
- For combining datasets with different variables, see merge.
- For combining datasets or data arrays with different indexes or missing values, see combine.
- For combining datasets or data arrays along multiple dimensions see combining.multi.
To combine arrays along existing or new dimension into a larger array, you can use :py~xarray.concat
. concat
takes an iterable of DataArray
or Dataset
objects, as well as a dimension name, and concatenates along that dimension:
python
- da = xr.DataArray(
np.arange(6).reshape(2, 3), [("x", ["a", "b"]), ("y", [10, 20, 30])]
) da.isel(y=slice(0, 1)) # same as da[:, :1] # This resembles how you would use np.concatenate: xr.concat([da[:, :1], da[:, 1:]], dim="y") # For more friendly pandas-like indexing you can use: xr.concat([da.isel(y=slice(0, 1)), da.isel(y=slice(1, None))], dim="y")
In addition to combining along an existing dimension, concat
can create a new dimension by stacking lower dimensional arrays together:
python
da.sel(x="a") xr.concat([da.isel(x=0), da.isel(x=1)], "x")
If the second argument to concat
is a new dimension name, the arrays will be concatenated along that new dimension, which is always inserted as the first dimension:
python
xr.concat([da.isel(x=0), da.isel(x=1)], "new_dim")
The second argument to concat
can also be an :py~pandas.Index
or :py~xarray.DataArray
object as well as a string, in which case it is used to label the values along the new dimension:
python
xr.concat([da.isel(x=0), da.isel(x=1)], pd.Index([-90, -100], name="new_dim"))
Of course, concat
also works on Dataset
objects:
python
ds = da.to_dataset(name="foo") xr.concat([ds.sel(x="a"), ds.sel(x="b")], "x")
:py~xarray.concat
has a number of options which provide deeper control over which variables are concatenated and how it handles conflicting variables between datasets. With the default parameters, xarray will load some coordinate variables into memory to compare them between datasets. This may be prohibitively expensive if you are manipulating your dataset lazily using dask
.
To combine variables and coordinates between multiple DataArray
and/or Dataset
objects, use :py~xarray.merge
. It can merge a list of Dataset
, DataArray
or dictionaries of objects convertible to DataArray
objects:
python
xr.merge([ds, ds.rename({"foo": "bar"})]) xr.merge([xr.DataArray(n, name="var%d" % n) for n in range(5)])
If you merge another dataset (or a dictionary including data array objects), by default the resulting dataset will be aligned on the union of all index coordinates:
python
other = xr.Dataset({"bar": ("x", [1, 2, 3, 4]), "x": list("abcd")}) xr.merge([ds, other])
This ensures that merge
is non-destructive. xarray.MergeError
is raised if you attempt to merge two variables with the same name but different values:
@verbatim In [1]: xr.merge([ds, ds + 1]) MergeError: conflicting values for variable 'foo' on objects to be combined: first value: <xarray.Variable (x: 2, y: 3)> array([[ 0.4691123 , -0.28286334, -1.5090585 ], [-1.13563237, 1.21211203, -0.17321465]]) second value: <xarray.Variable (x: 2, y: 3)> array([[ 1.4691123 , 0.71713666, -0.5090585 ], [-0.13563237, 2.21211203, 0.82678535]])
The same non-destructive merging between DataArray
index coordinates is used in the :py~xarray.Dataset
constructor:
python
xr.Dataset({"a": da.isel(x=slice(0, 1)), "b": da.isel(x=slice(1, 2))})
The instance method :py~xarray.DataArray.combine_first
combines two datasets/data arrays and defaults to non-null values in the calling object, using values from the called object to fill holes. The resulting coordinates are the union of coordinate labels. Vacant cells as a result of the outer-join are filled with NaN
. For example:
python
ar0 = xr.DataArray([[0, 0], [0, 0]], [("x", ["a", "b"]), ("y", [-1, 0])]) ar1 = xr.DataArray([[1, 1], [1, 1]], [("x", ["b", "c"]), ("y", [0, 1])]) ar0.combine_first(ar1) ar1.combine_first(ar0)
For datasets, ds0.combine_first(ds1)
works similarly to xr.merge([ds0, ds1])
, except that xr.merge
raises MergeError
when there are conflicting values in variables to be merged, whereas .combine_first
defaults to the calling object's values.
In contrast to merge
, :py~xarray.Dataset.update
modifies a dataset in-place without checking for conflicts, and will overwrite any existing variables with new values:
python
ds.update({"space": ("space", [10.2, 9.4, 3.9])})
However, dimensions are still required to be consistent between different Dataset variables, so you cannot change the size of a dimension unless you replace all dataset variables that use it.
update
also performs automatic alignment if necessary. Unlike merge
, it maintains the alignment of the original array instead of merging indexes:
python
ds.update(other)
The exact same alignment logic when setting a variable with __setitem__
syntax:
python
ds["baz"] = xr.DataArray([9, 9, 9, 9, 9], coords=[("x", list("abcde"))]) ds.baz
Xarray objects can be compared by using the :py~xarray.Dataset.equals
, :py~xarray.Dataset.identical
and :py~xarray.Dataset.broadcast_equals
methods. These methods are used by the optional compat
argument on concat
and merge
.
:py~xarray.Dataset.equals
checks dimension names, indexes and array values:
python
da.equals(da.copy())
:py~xarray.Dataset.identical
also checks attributes, and the name of each object:
python
da.identical(da.rename("bar"))
:py~xarray.Dataset.broadcast_equals
does a more relaxed form of equality check that allows variables to have different dimensions, as long as values are constant along those new dimensions:
python
left = xr.Dataset(coords={"x": 0}) right = xr.Dataset({"x": [0, 0, 0]}) left.broadcast_equals(right)
Like pandas objects, two xarray objects are still equal or identical if they have missing values marked by NaN
in the same locations.
In contrast, the ==
operation performs element-wise comparison (like numpy):
python
da == da.copy()
Note that NaN
does not compare equal to NaN
in element-wise comparison; you may need to deal with missing values explicitly.
The compat
argument 'no_conflicts'
is only available when combining xarray objects with merge
. In addition to the above comparison methods it allows the merging of xarray objects with locations where either have NaN
values. This can be used to combine data with overlapping coordinates as long as any non-missing values agree or are disjoint:
python
ds1 = xr.Dataset({"a": ("x", [10, 20, 30, np.nan])}, {"x": [1, 2, 3, 4]}) ds2 = xr.Dataset({"a": ("x", [np.nan, 30, 40, 50])}, {"x": [2, 3, 4, 5]}) xr.merge([ds1, ds2], compat="no_conflicts")
Note that due to the underlying representation of missing values as floating point numbers (NaN
), variable data type is not always preserved when merging in this manner.
For combining many objects along multiple dimensions xarray provides :py~xarray.combine_nested
and :py~xarray.combine_by_coords
. These functions use a combination of concat
and merge
across different variables to combine many objects into one.
:py~xarray.combine_nested
requires specifying the order in which the objects should be combined, while :py~xarray.combine_by_coords
attempts to infer this ordering automatically from the coordinates in the data.
:py~xarray.combine_nested
is useful when you know the spatial relationship between each object in advance. The datasets must be provided in the form of a nested list, which specifies their relative position and ordering. A common task is collecting data from a parallelized simulation where each processor wrote out data to a separate file. A domain which was decomposed into 4 parts, 2 each along both the x and y axes, requires organising the datasets into a doubly-nested list, e.g:
python
- arr = xr.DataArray(
name="temperature", data=np.random.randint(5, size=(2, 2)), dims=["x", "y"]
) arr ds_grid = [[arr, arr], [arr, arr]] xr.combine_nested(ds_grid, concat_dim=["x", "y"])
:py~xarray.combine_nested
can also be used to explicitly merge datasets with different variables. For example if we have 4 datasets, which are divided along two times, and contain two different variables, we can pass None
to 'concat_dim'
to specify the dimension of the nested list over which we wish to use merge
instead of concat
:
python
temp = xr.DataArray(name="temperature", data=np.random.randn(2), dims=["t"]) precip = xr.DataArray(name="precipitation", data=np.random.randn(2), dims=["t"]) ds_grid = [[temp, precip], [temp, precip]] xr.combine_nested(ds_grid, concat_dim=["t", None])
:py~xarray.combine_by_coords
is for combining objects which have dimension coordinates which specify their relationship to and order relative to one another, for example a linearly-increasing 'time' dimension coordinate.
Here we combine two datasets using their common dimension coordinates. Notice they are concatenated in order based on the values in their dimension coordinates, not on their position in the list passed to combine_by_coords
.
python
x1 = xr.DataArray(name="foo", data=np.random.randn(3), coords=[("x", [0, 1, 2])]) x2 = xr.DataArray(name="foo", data=np.random.randn(3), coords=[("x", [3, 4, 5])]) xr.combine_by_coords([x2, x1])
These functions can be used by :py~xarray.open_mfdataset
to open many files as one dataset. The particular function used is specified by setting the argument 'combine'
to 'by_coords'
or 'nested'
. This is useful for situations where your data is split across many files in multiple locations, which have some known relationship between one another.