Error when Running SUEWS Advance in QGIS

[ozyaz]

Describe the bug

Dear SUEWS Community,

I want to use the SUEWS for my study area in Berlin. Before that I am trying to run a test version so I know that my data is accurate. So it is only 500mx500m area with 121 Grids. SUEWS prepared worked, but for SUEWS advanced either my QGIS crashed or it gave an

Error:
Unable to allocate 460. MiB for an array with shape (105120,573) and data type float64

To Reproduce
the following output files are from SUEWS Prepare. So in the Advance window, as an input, the folder containing these files should be given and run.

SUEWS-Advance settings are as recommended:

SUEWS Prepare Output files:
SUEWS_AnthropogenicEmission.txt
SUEWS_BiogenCO2.txt
SUEWS_Conductance.txt
SUEWS_ESTMCoefficients.txt
SUEWS_Irrigation.txt
SUEWS_NonVeg.txt
SUEWS_OHMCoefficients.txt
SUEWS_Profiles.txt
SUEWS_SiteSelect.txt
SUEWS_Snow.txt
SUEWS_Soil.txt
SUEWS_Veg.txt
SUEWS_Water.txt
SUEWS_WithinGridWaterDist.txt
ber_2019_data_60.txt

Screenshots
ERROR screen

Additional context
I did create my own land cover according to the SUEWS fractions, all rasters I had are with 1m resolution.

• System Info
  CPU Type: x86_64
  Kernel Type: winnt
  Kernel Version: 10.0.22000

Version and platform (please complete the following information):

• SUEWS version: v2020a
• OS: Windows 11
• QGIS Version: 3.16.16 Hannover

[sunt05]

Sorry to see the error, which is not uncommon for UMEP users: this looks to be a memory issue that is beyond our control.

How much memory does your machine have?
After getting the input files prepared by SUEWS prepare, did you try to run the SUEWS simulations from the fresh QGIS (i.e. restart QGIS)?

Or, if you are OK with SuPy, please try out your case with SuPy directly – the actual simulation engine behind the UMEP SUEWS utility: https://supy.readthedocs.io/en/latest/

[ozyaz]

Thank you for your feedback, I actually tried the same input data with Supy. It gave the same error, even though I changed the computer I am working with. There shouldnt be a problem with memory ?!(total RAM is 32 GB). Do you think it is because of the dataset? I havent found any guide or recommendation of the study area size, like how many grids should we work with? I have 121 Grids (500x500m) Is that a lot?

The Error looks like:

[sunt05]

Sorry to learn the issue persists.

Can you please try the following - as the array numpy tried to allocate is not large at all (i.e. 460 MB) for a machine with 32GB RAM, I'm wondering if you are using a 32 bit system:

sp.show_versions()

and paste the whole output here.

Thanks.

[ozyaz]

it doesnt seems like that's the problem ->

SuPy versions
-------------
supy: 2022.4.7 
supy_driver: 2021a5

=================
SYSTEM DEPENDENCY
C:\Users\Yazici\anaconda3\envs\SUEWS\lib\site-packages\_distutils_hack\__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")

INSTALLED VERSIONS
------------------
commit           : e8093ba372f9adfe79439d90fe74b0b5b6dea9d6
python           : 3.9.12.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
Version          : 10.0.19044
machine          : AMD64
processor        : Intel64 Family 6 Model 45 Stepping 7, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : de_DE.cp1252

pandas           : 1.4.3
numpy            : 1.23.1
pytz             : 2022.1
dateutil         : 2.8.2
setuptools       : 61.2.0
pip              : 22.1.2
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 3.0.3
IPython          : 8.4.0
pandas_datareader: None
bs4              : 4.11.1
bottleneck       : None
brotli           : None
fastparquet      : None
fsspec           : 2022.5.0
gcsfs            : None
markupsafe       : 2.1.1
matplotlib       : 3.5.2
numba            : None
numexpr          : 2.8.3
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pyreadstat       : None
pyxlsb           : None
s3fs             : None
scipy            : 1.8.1
snappy           : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : 2022.3.0
xlrd             : None
xlwt             : None
zstandard        : None

[sunt05]

So it seems the numpy in your env failed to request enough memory.
This might be a minimal case to reproduce the error:

import numpy as np
ar=np.full((105118,573),np.pi)

If this code fails, then you need to find out why your numpy/python behaves so.

---

However, as a workaround, you can split the case into smaller chunks - either shortening the simulation period or reducing number of grids - and save the respective output pieces as external files (e.g. pickle files); later when you enter the analysis stage, load them back selectively to reduce the memory load.

[ozyaz]

Thank you, I did change my meteo data to one month instead of a year and it worked.