Error in building the network for the first time

[EmreYorat]

Hello,
I and my Pypsa Middle-East Teammates have followed the instructions to install pypsa-africa and dowloaded the data but the code -snakemake -j 1 networks/elec_s_6.nc --forceall- for building the network for the first time is giving us error in rule build_shapes which directs to a script in anaconda3/envs/pypsa-africa/lib/python3.9/site-packages/xarray/backends/plugins.py. I am sharing the error log below. If any one has suggestion(s) to overcome this problem we will be appreciate it. Thank you.
error.txt

[ekatef]

@E-Yorat, thank you for opening the issue.

I'm able to reproduce it on original data for Nigeria, too. As far as I can trace the produced error, it is triggered with the GDP_dataset = xr.open_dataset(GDP_nc) line in build shapes.py.

The nc file itself seems to be alright; playing with build_shape_options in config.yaml does not change anything. Explicit specifying the engine xr.open_dataset(GDP_nc, engine = "netcdf4") was also not a solution.

Probably, that problem connected with this xarray issue which was fixed recently.

Upd: Update of the xarray in the conda virtual environment has resolved the issue with network building (conda update xarray).

[oayana]

@E-Yorat, thank you for opening the issue.

I'm able to reproduce it on original data for Nigeria, too. As far as I can trace the produced error, it is triggered with the GDP_dataset = xr.open_dataset(GDP_nc) line in build shapes.py.

The nc file itself seems to be alright; playing with build_shape_options in config.yaml does not change anything. Explicit specifying the engine xr.open_dataset(GDP_nc, engine = "netcdf4") was also not a solution.

Probably, that problem connected with this xarray issue which was fixed recently.

Upd: Update of the xarray in the conda virtual environment has resolved the issue with network building (conda update xarray).

Thank you for your suggestion. Problem solved with "conda update xarray".

[ekatef]

@oayana Great to hear that it has helped. Thank you for letting know.

[pz-max]

Great @oayana, @ekatef and @E-Yorat for opening and solving the issue 👍

[davide-f]

Sorry for temporary reopening the issue, but I would like to point out that to avoid future problems related to that, we may temporarily specify exactly the versions of the packages that we are using.
The problems above, in fact, were crearly related to different versions of the packages and to avoid such issues in the features, we may fix the environment packages for the current development; once the code is stable, we may reduce the binds, what do you think?
@pz-max for example, originally in the "toast" environment, we have been using python 3.8, but in subsequent changes that requirement has been removed as well as the versions of the packages is now not constrained, which may generate future errors and ambiguities. It may be a good idea to use a common environment with the same versions for everyone, what do you think?

[ekatef]

@davide-f I think it would be a very good idea to have a versions specification for all the dependencies. And could it probably be kept in the stable releases, too? According to my user experience, compatibility is one of biggest issues when you are trying to guess how to use a new computational tool :)

[pz-max]

I think if we have the CI most such issues are fixed because tests are running once or twice a day @davide-f
For me, it also sounds very good to have "tagged" releases with fixed environments. Now until pypsa-africa solves we work on v0.1.0 :)

[EmreYorat]

@davide-f That would make sense because as new users want to try the software for the first time it would be a good impression if it works right away otherwise trying to solve the issues to make it work that may discourage people to use the software at the beginning.
I also thank to @ekatef for solving our problem.

[pz-max]

I work now on the continuous integration test so we solve that issue that code does not work.
Thanks @E-Yorat and @ekatef for sharing your thoughts.

[davide-f]

@pz-max I can prepare the environment of the "toast" version built on python 3.8. Do you agree?
Actually, the issue and fix described above do not work on my linux machine; do yours work on linux?

[pz-max]

Hi @davide-f,
My code only breaks now somewhere at the clustering. Debugging at the moment.
I would suggest to remove toast conda env remove --name toast
and install the environment with all updates/resolves conda env create -f envs/environment.yaml