Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sza_check from trollflow2 fails with KeyError: 'start_time' #1943

Closed
jdaniel006 opened this issue Dec 16, 2021 · 12 comments
Closed

sza_check from trollflow2 fails with KeyError: 'start_time' #1943

jdaniel006 opened this issue Dec 16, 2021 · 12 comments
Assignees
@mraspaud
Labels
bug

Comments

@jdaniel006
Copy link

jdaniel006 commented Dec 16, 2021 •
edited

Describe the bug
I have implemented a trollflow2 chain based on the Docker example to process MSG SEVIRI data in native format downloaded from Eumetsat Data Store. When I first used docker on another machine a month ago, everything was working perfect.

Then, a week ago, I replicated the structure on another machine but without docker. I.e., I followed the docker build file as a recipe. After the installation, the same files started failing to be processed due to an error while Sun Zenit Angle is being checked. It seems that the function is getting an empty scene, as dumping the contents of the snc variable yields an empty dictionary.

If sza_check is removed from the trolflow2.ini config file, files are processed.

Comparing both machines, it seems that satpy 0.32.0 (not been able to test with 0.33.0) breaks something, as downgrading to satpy 0.31.0 solves the problem

To Reproduce

Just make an installation following Dockerfile_stable build_dir (either as a container or standalone)

Expected behavior
Trollflow2 successfully cheks sza

Actual results
Text output of actual results or error messages including full tracebacks if applicable.
2021-12-03 19:59:40,706 DEBG 'trollflow2' stdout output:
[INFO: 2021-12-03 19:59:40 : posttroll.subscriber] Subscriber adding address tcp://172.31.134.20:43975 with topics ['pytroll://dataset/seviri_native']

2021-12-03 19:59:40,709 DEBG 'trollflow2' stdout output:
[INFO: 2021-12-03 19:59:40 : trollflow2.plugins] Generating scene

2021-12-03 19:59:40,709 DEBG 'trollflow2' stdout output:
[DEBUG: 2021-12-03 19:59:40 : satpy.readers.yaml_reader] Reading ('/home/meteo/.conda/envs/pytroll/lib/python3.9/site-packages/satpy/etc/readers/seviri_l1b_native.yaml',)

2021-12-03 19:59:40,753 DEBG 'trollflow2' stdout output:
[DEBUG: 2021-12-03 19:59:40 : satpy.readers.yaml_reader] Assigning to seviri_l1b_native: ['/home/meteo/METEOSAT/temp/MSG4-SEVI-MSG15-0100-NA-20211202101242.230000000Z-NA.nat']

2021-12-03 19:59:40,801 DEBG 'trollflow2' stdout output:
[ERROR: 2021-12-03 19:59:40 : trollflow2.launcher] Process crashed
Traceback (most recent call last):
File "/home/meteo/.conda/envs/pytroll/lib/python3.9/site-packages/trollflow2/launcher.py", line 385, in process
cwrk.pop('fun')(job, **cwrk)
File "/home/meteo/.conda/envs/pytroll/lib/python3.9/site-packages/trollflow2/plugins/init.py", line 514, in sza_check
start_time = scn.attrs['start_time']
KeyError: 'start_time'

2021-12-03 19:59:40,801 DEBG 'trollflow2' stdout output:
[DEBUG: 2021-12-03 19:59:40 : trollflow2.launcher] Cleaning up

2021-12-03 19:59:40,801 DEBG 'trollflow2' stdout output:
[DEBUG: 2021-12-03 19:59:40 : posttroll.publisher] exiting publish

2021-12-03 19:59:40,842 DEBG 'trollflow2' stdout output:
Process SpawnProcess-2:

2021-12-03 19:59:40,843 DEBG 'trollflow2' stdout output:
Traceback (most recent call last):
File "/home/meteo/.conda/envs/pytroll/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/meteo/.conda/envs/pytroll/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)

2021-12-03 19:59:40,843 DEBG 'trollflow2' stdout output:
File "/home/meteo/.conda/envs/pytroll/lib/python3.9/site-packages/trollflow2/launcher.py", line 335, in queue_logged_process
process(msg, prod_list, produced_files)
File "/home/meteo/.conda/envs/pytroll/lib/python3.9/site-packages/trollflow2/launcher.py", line 385, in process
cwrk.pop('fun')(job, **cwrk)
File "/home/meteo/.conda/envs/pytroll/lib/python3.9/site-packages/trollflow2/plugins/init.py", line 514, in sza_check
start_time = scn.attrs['start_time']
KeyError: 'start_time'

2021-12-03 19:59:40,844 DEBG 'trollflow2' stdout output:
[DEBUG: 2021-12-03 19:59:40 : posttroll.publisher] exiting publish

Environment Info:

NOT WORKING

packages in environment at /home/meteo/.conda/envs/pytroll:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
affine 2.3.0 py_0 conda-forge
aggdraw 1.3.12 py39h54733d6_3 conda-forge
aiobotocore 2.0.1 pypi_0 pypi
aiohttp 3.8.1 pypi_0 pypi
aioitertools 0.8.0 pypi_0 pypi
aiosignal 1.2.0 pypi_0 pypi
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
asciitree 0.3.3 pypi_0 pypi
async-timeout 4.0.1 pypi_0 pypi
attrs 21.2.0 pyhd8ed1ab_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
bcrypt 3.2.0 py39h3811e60_2 conda-forge
blosc 1.21.0 h9c3ff4c_0 conda-forge
bokeh 2.4.2 py39hf3d152e_0 conda-forge
boost-cpp 1.74.0 h359cf19_5 conda-forge
botocore 1.22.8 pypi_0 pypi
brotlipy 0.7.0 py39h3811e60_1003 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
ca-certificates 2021.10.8 ha878542_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cairo 1.16.0 ha00ac49_1009 conda-forge
certifi 2021.10.8 pypi_0 pypi
cffi 1.15.0 py39h4bc2ebd_0 conda-forge
cfitsio 4.0.0 h9a35b8e_0 conda-forge
cftime 1.5.1.1 py39hce5d2b2_1 conda-forge
charset-normalizer 2.0.9 pyhd8ed1ab_0 conda-forge
click 8.0.3 py39hf3d152e_1 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge
colorama 0.4.4 pyh9f0ad1d_0 conda-forge
configobj 5.0.6 pypi_0 pypi
cryptography 36.0.0 py39h95dcef6_0 conda-forge
curl 7.80.0 h2574ce0_0 conda-forge
cytoolz 0.11.2 py39h3811e60_1 conda-forge
dask 2021.11.2 pyhd8ed1ab_0 conda-forge
dask-core 2021.11.2 pyhd8ed1ab_0 conda-forge
decorator 5.1.0 pyhd8ed1ab_0 conda-forge
distributed 2021.11.2 py39hf3d152e_0 conda-forge
docutils 0.18.1 py39hf3d152e_0 conda-forge
donfig 0.6.0 pyhd8ed1ab_0 conda-forge
dpath 1.5.0 pypi_0 pypi
expat 2.4.1 h9c3ff4c_0 conda-forge
fasteners 0.16.3 pypi_0 pypi
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.13.1 hba837de_1005 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
freexl 1.0.6 h7f98852_0 conda-forge
frozenlist 1.2.0 pypi_0 pypi
fsspec 2021.11.1 pyhd8ed1ab_0 conda-forge
gdal 3.4.0 py39h0494519_12 conda-forge
geos 3.10.1 h9c3ff4c_1 conda-forge
geotiff 1.7.0 h90a4e78_5 conda-forge
gettext 0.19.8.1 h73d1719_1008 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
h5netcdf 0.11.0 pyhd8ed1ab_0 conda-forge
h5py 3.6.0 nompi_py39h7e08c79_100 conda-forge
hdf4 4.2.15 h10796ff_3 conda-forge
hdf5 1.12.1 nompi_h2750804_103 conda-forge
heapdict 1.0.1 py_0 conda-forge
icu 69.1 h9c3ff4c_0 conda-forge
idna 3.3 pypi_0 pypi
importlib-metadata 4.8.2 py39hf3d152e_0 conda-forge
importlib_metadata 4.8.2 hd8ed1ab_0 conda-forge
inotify 0.2.10 pypi_0 pypi
ipython 7.30.1 py39hf3d152e_0 conda-forge
jbig 2.1 h7f98852_2003 conda-forge
jedi 0.18.1 py39hf3d152e_0 conda-forge
jinja2 3.0.3 pyhd8ed1ab_0 conda-forge
jmespath 0.10.0 pypi_0 pypi
jpeg 9d h36c2ea0_0 conda-forge
json-c 0.15 h98cffda_0 conda-forge
kealib 1.4.14 h87e4c3c_3 conda-forge
krb5 1.19.2 hcc1bbae_3 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
lerc 3.0 h9c3ff4c_0 conda-forge
libblas 3.9.0 12_linux64_openblas conda-forge
libcblas 3.9.0 12_linux64_openblas conda-forge
libcurl 7.80.0 h2574ce0_0 conda-forge
libdap4 3.20.6 hd7c4107_2 conda-forge
libdeflate 1.8 h7f98852_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 11.2.0 h1d223b6_11 conda-forge
libgdal 3.4.0 hbe510e8_12 conda-forge
libgfortran-ng 11.2.0 h69a702a_11 conda-forge
libgfortran5 11.2.0 h5c6108e_11 conda-forge
libglib 2.70.2 h174f98d_0 conda-forge
libgomp 11.2.0 h1d223b6_11 conda-forge
libiconv 1.16 h516909a_0 conda-forge
libkml 1.3.0 h238a007_1014 conda-forge
liblapack 3.9.0 12_linux64_openblas conda-forge
libnetcdf 4.8.1 nompi_hb3fd0d9_101 conda-forge
libnghttp2 1.43.0 h812cca2_1 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.18 pthreads_h8fe5266_0 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libpq 14.1 hd57d9b9_1 conda-forge
librttopo 1.1.0 h30b5eef_8 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libspatialite 5.0.1 hf3ee066_12 conda-forge
libssh2 1.10.0 ha56f1ee_2 conda-forge
libstdcxx-ng 11.2.0 he4da1e4_11 conda-forge
libtiff 4.3.0 h6f004c6_2 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libwebp-base 1.2.1 h7f98852_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxml2 2.9.12 h885dcf4_1 conda-forge
libzip 1.8.0 h4de3113_1 conda-forge
libzlib 1.2.11 h36c2ea0_1013 conda-forge
locket 0.2.1 pypi_0 pypi
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
markupsafe 2.0.1 py39h3811e60_1 conda-forge
matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge
meld3 2.0.1 pyh9f0ad1d_0 conda-forge
monotonic 1.5 py_0 conda-forge
msgpack-python 1.0.3 py39h1a9c180_0 conda-forge
multidict 5.2.0 pypi_0 pypi
ncurses 6.2 h58526e2_4 conda-forge
netcdf4 1.5.8 nompi_py39h64b754b_101 conda-forge
netifaces 0.11.0 pypi_0 pypi
nose 1.3.7 pypi_0 pypi
nspr 4.32 h9c3ff4c_1 conda-forge
nss 3.73 hb5efdd6_0 conda-forge
numcodecs 0.9.1 py39he80948d_2 conda-forge
numpy 1.21.4 py39hdbf815f_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 1.1.1l h7f98852_0 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.3.4 pypi_0 pypi
paramiko 2.8.1 pyhd8ed1ab_0 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
partd 1.2.0 pyhd8ed1ab_0 conda-forge
pcre 8.45 h9c3ff4c_0 conda-forge
pexpect 4.8.0 pyh9f0ad1d_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 8.4.0 py39ha612740_0 conda-forge
pip 21.3.1 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
pooch 1.5.2 pyhd8ed1ab_0 conda-forge
poppler 21.11.0 ha39eefc_0 conda-forge
poppler-data 0.4.11 hd8ed1ab_0 conda-forge
postgresql 14.1 h2510834_1 conda-forge
posttroll 1.6.1 pypi_0 pypi
proj 8.2.0 h277dcde_0 conda-forge
prompt-toolkit 3.0.23 pyha770c72_0 conda-forge
psutil 5.8.0 py39h3811e60_2 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pycoast 1.5.0 pyhd8ed1ab_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pygments 2.10.0 pyhd8ed1ab_0 conda-forge
pyinotify 0.9.6 py39hf3d152e_1004 conda-forge
pykdtree 1.3.4 py39hce5d2b2_1 conda-forge
pynacl 1.4.0 py39h3811e60_3 conda-forge
pyopenssl 21.0.0 pyhd8ed1ab_0 conda-forge
pyorbital 1.6.1 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.6 pyhd8ed1ab_0 conda-forge
pyproj 3.3.0 py39h1b810de_0 conda-forge
pyresample 1.22.3 py39hde0f152_0 conda-forge
pyshp 2.1.3 pyh44b312d_0 conda-forge
pysocks 1.7.1 py39hf3d152e_4 conda-forge
pyspectral 0.10.5 pyhd8ed1ab_0 conda-forge
python 3.9.7 hb7a2778_3_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-geotiepoints 1.3.0 py39hce5d2b2_1 conda-forge
python_abi 3.9 2_cp39 conda-forge
pytroll-collectors 0.11.1 pypi_0 pypi
pytroll-schedule 0.5.2 pypi_0 pypi
pytz 2021.3 pyhd8ed1ab_0 conda-forge
pyyaml 6.0 py39h3811e60_3 conda-forge
pyzmq 22.3.0 py39h37b5a0c_1 conda-forge
rasterio 1.2.10 py39he0fb565_3 conda-forge
readline 8.1 h46c0cb4_0 conda-forge
requests 2.26.0 pyhd8ed1ab_1 conda-forge
s3fs 2021.11.1 pypi_0 pypi
satpy 0.32.0 pyhd8ed1ab_0 conda-forge
scipy 1.7.3 py39hee8e79c_0 conda-forge
scp 0.14.1 pyhd8ed1ab_0 conda-forge
setuptools 59.5.0 pypi_0 pypi
six 1.16.0 pyh6c4a22f_0 conda-forge
snuggs 1.4.7 py_0 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
sqlite 3.37.0 h9cd32fc_0 conda-forge
superlance 1.1.0.dev0 pypi_0 pypi
supervisor 4.3.0.dev0 pypi_0 pypi
tblib 1.7.0 pyhd8ed1ab_0 conda-forge
tiledb 2.5.2 h2038895_0 conda-forge
tk 8.6.11 h27826a3_1 conda-forge
toolz 0.11.2 pyhd8ed1ab_0 conda-forge
tornado 6.1 py39h3811e60_2 conda-forge
tqdm 4.62.3 pyhd8ed1ab_0 conda-forge
traitlets 5.1.1 pyhd8ed1ab_0 conda-forge
trollflow2 0.11.1 pypi_0 pypi
trollimage 1.17.0 pyhd8ed1ab_0 conda-forge
trollmoves 0.9.0 pypi_0 pypi
trollsift 0.3.5 pyh44b312d_0 conda-forge
typing_extensions 4.0.1 pyha770c72_0 conda-forge
tzcode 2021e h7f98852_0 conda-forge
tzdata 2021e he74cb21_0 conda-forge
urllib3 1.26.7 pyhd8ed1ab_0 conda-forge
watchdog 2.1.6 pypi_0 pypi
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
wheel 0.37.0 pyhd8ed1ab_1 conda-forge
wrapt 1.13.3 pypi_0 pypi
xarray 0.20.1 pyhd8ed1ab_0 conda-forge
xerces-c 3.2.3 h8ce2273_4 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.2 h7f98852_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
yarl 1.7.2 pypi_0 pypi
zarr 2.10.3 pyhd8ed1ab_0 conda-forge
zeromq 4.3.4 h9c3ff4c_1 conda-forge
zict 2.0.0 py_0 conda-forge
zipp 3.6.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.11 h36c2ea0_1013 conda-forge
zstd 1.5.0 ha95c52a_0 conda-forge

WORKING

packages in environment at /home/meteo/.conda/envs/pytroll:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
affine 2.3.0 py_0 conda-forge
aggdraw 1.3.12 py39h54733d6_3 conda-forge
aiobotocore 2.0.1 pypi_0 pypi
aiohttp 3.8.1 pypi_0 pypi
aioitertools 0.8.0 pypi_0 pypi
aiosignal 1.2.0 pypi_0 pypi
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
asciitree 0.3.3 pypi_0 pypi
async-timeout 4.0.1 pypi_0 pypi
attrs 21.2.0 pyhd8ed1ab_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
bcrypt 3.2.0 py39h3811e60_2 conda-forge
blosc 1.21.0 h9c3ff4c_0 conda-forge
bokeh 2.4.2 py39hf3d152e_0 conda-forge
boost-cpp 1.74.0 h359cf19_5 conda-forge
botocore 1.22.8 pypi_0 pypi
brotlipy 0.7.0 py39h3811e60_1003 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
ca-certificates 2021.10.8 ha878542_0 conda-forge
cached-property 1.5.2 hd8ed1ab_1 conda-forge
cached_property 1.5.2 pyha770c72_1 conda-forge
cairo 1.16.0 ha00ac49_1009 conda-forge
certifi 2021.10.8 pypi_0 pypi
cffi 1.15.0 py39h4bc2ebd_0 conda-forge
cfitsio 4.0.0 h9a35b8e_0 conda-forge
cftime 1.5.1.1 py39hce5d2b2_1 conda-forge
charset-normalizer 2.0.9 pyhd8ed1ab_0 conda-forge
click 8.0.3 py39hf3d152e_1 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge
colorama 0.4.4 pyh9f0ad1d_0 conda-forge
configobj 5.0.6 pypi_0 pypi
cryptography 36.0.0 py39h9ce1e76_0
curl 7.80.0 h494985f_1 conda-forge
cytoolz 0.11.2 py39h3811e60_1 conda-forge
dask 2021.12.0 pyhd8ed1ab_0 conda-forge
dask-core 2021.12.0 pyhd8ed1ab_0 conda-forge
decorator 5.1.0 pyhd8ed1ab_0 conda-forge
distributed 2021.12.0 py39hf3d152e_0 conda-forge
docutils 0.18.1 py39hf3d152e_0 conda-forge
donfig 0.6.0 pyhd8ed1ab_0 conda-forge
dpath 1.5.0 pypi_0 pypi
expat 2.4.1 h9c3ff4c_0 conda-forge
fasteners 0.16.3 pypi_0 pypi
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.13.1 hba837de_1005 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
freexl 1.0.6 h7f98852_0 conda-forge
frozenlist 1.2.0 pypi_0 pypi
fsspec 2021.11.1 pyhd8ed1ab_0 conda-forge
gdal 3.4.0 py39he073ec8_12 conda-forge
geos 3.10.1 h9c3ff4c_1 conda-forge
geotiff 1.7.0 h90a4e78_5 conda-forge
gettext 0.19.8.1 h73d1719_1008 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
h5netcdf 0.11.0 pyhd8ed1ab_0 conda-forge
h5py 3.6.0 nompi_py39h7e08c79_100 conda-forge
hdf4 4.2.15 h10796ff_3 conda-forge
hdf5 1.12.1 nompi_h7f166f4_103 conda-forge
heapdict 1.0.1 py_0 conda-forge
icu 69.1 h9c3ff4c_0 conda-forge
idna 3.3 pypi_0 pypi
importlib-metadata 4.8.2 py39hf3d152e_0 conda-forge
importlib_metadata 4.8.2 hd8ed1ab_0 conda-forge
inotify 0.2.10 pypi_0 pypi
ipython 7.30.1 py39hf3d152e_0 conda-forge
jbig 2.1 h7f98852_2003 conda-forge
jedi 0.18.1 py39hf3d152e_0 conda-forge
jinja2 3.0.3 pyhd8ed1ab_0 conda-forge
jmespath 0.10.0 pypi_0 pypi
jpeg 9d h36c2ea0_0 conda-forge
json-c 0.15 h98cffda_0 conda-forge
kealib 1.4.14 h87e4c3c_3 conda-forge
krb5 1.19.2 h48eae69_3 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
lerc 3.0 h9c3ff4c_0 conda-forge
libblas 3.9.0 12_linux64_openblas conda-forge
libcblas 3.9.0 12_linux64_openblas conda-forge
libcurl 7.80.0 h494985f_1 conda-forge
libdap4 3.20.6 hd7c4107_2 conda-forge
libdeflate 1.8 h7f98852_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 11.2.0 h1d223b6_11 conda-forge
libgdal 3.4.0 h1504ab5_12 conda-forge
libgfortran-ng 11.2.0 h69a702a_11 conda-forge
libgfortran5 11.2.0 h5c6108e_11 conda-forge
libglib 2.70.2 h174f98d_0 conda-forge
libgomp 11.2.0 h1d223b6_11 conda-forge
libiconv 1.16 h516909a_0 conda-forge
libkml 1.3.0 h238a007_1014 conda-forge
liblapack 3.9.0 12_linux64_openblas conda-forge
libnetcdf 4.8.1 nompi_hb3fd0d9_101 conda-forge
libnghttp2 1.43.0 ha19adfc_1 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.18 pthreads_h8fe5266_0 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libpq 14.1 h676c864_1 conda-forge
librttopo 1.1.0 h30b5eef_8 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libspatialite 5.0.1 hf3ee066_12 conda-forge
libssh2 1.10.0 ha35d2d1_2 conda-forge
libstdcxx-ng 11.2.0 he4da1e4_11 conda-forge
libtiff 4.3.0 h6f004c6_2 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libwebp-base 1.2.1 h7f98852_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxml2 2.9.12 h885dcf4_1 conda-forge
libzip 1.8.0 h1c5bbd1_1 conda-forge
libzlib 1.2.11 h36c2ea0_1013 conda-forge
locket 0.2.1 pypi_0 pypi
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
markupsafe 2.0.1 py39h3811e60_1 conda-forge
matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge
meld3 2.0.1 pyh9f0ad1d_0 conda-forge
monotonic 1.5 py_0 conda-forge
msgpack-python 1.0.3 py39h1a9c180_0 conda-forge
multidict 5.2.0 pypi_0 pypi
ncurses 6.2 h58526e2_4 conda-forge
netcdf4 1.5.8 nompi_py39h64b754b_101 conda-forge
netifaces 0.11.0 pypi_0 pypi
nose 1.3.7 pypi_0 pypi
nspr 4.32 h9c3ff4c_1 conda-forge
nss 3.73 hb5efdd6_0 conda-forge
numcodecs 0.9.1 py39he80948d_2 conda-forge
numpy 1.21.4 py39hdbf815f_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 3.0.0 h7f98852_2 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.3.4 pypi_0 pypi
paramiko 2.8.1 pyhd8ed1ab_0 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
partd 1.2.0 pyhd8ed1ab_0 conda-forge
pcre 8.45 h9c3ff4c_0 conda-forge
pexpect 4.8.0 pyh9f0ad1d_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 8.4.0 py39ha612740_0 conda-forge
pip 21.3.1 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
pooch 1.5.2 pyhd8ed1ab_0 conda-forge
poppler 21.11.0 ha39eefc_0 conda-forge
poppler-data 0.4.11 hd8ed1ab_0 conda-forge
postgresql 14.1 hce44dc1_1 conda-forge
posttroll 1.6.1 pypi_0 pypi
proj 8.2.0 h277dcde_0 conda-forge
prompt-toolkit 3.0.24 pyha770c72_0 conda-forge
psutil 5.8.0 py39h3811e60_2 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pycoast 1.5.0 pyhd8ed1ab_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pygments 2.10.0 pyhd8ed1ab_0 conda-forge
pyinotify 0.9.6 py39hf3d152e_1004 conda-forge
pykdtree 1.3.4 py39hce5d2b2_1 conda-forge
pynacl 1.4.0 py39h3811e60_3 conda-forge
pyopenssl 21.0.0 pyhd8ed1ab_0 conda-forge
pyorbital 1.6.1 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.6 pyhd8ed1ab_0 conda-forge
pyproj 3.3.0 py39h1b810de_0 conda-forge
pyresample 1.22.3 py39hde0f152_0 conda-forge
pyshp 2.1.3 pyh44b312d_0 conda-forge
pysocks 1.7.1 py39hf3d152e_4 conda-forge
pyspectral 0.10.5 pyhd8ed1ab_0 conda-forge
python 3.9.7 hf930737_3_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-geotiepoints 1.3.0 py39hce5d2b2_1 conda-forge
python_abi 3.9 2_cp39 conda-forge
pytroll-collectors 0.11.1 pypi_0 pypi
pytroll-schedule 0.5.2 pypi_0 pypi
pytz 2021.3 pyhd8ed1ab_0 conda-forge
pyyaml 6.0 py39h3811e60_3 conda-forge
pyzmq 22.3.0 py39h37b5a0c_1 conda-forge
rasterio 1.2.10 py39he0fb565_3 conda-forge
readline 8.1 h46c0cb4_0 conda-forge
requests 2.26.0 pyhd8ed1ab_1 conda-forge
s3fs 2021.11.1 pypi_0 pypi
satpy 0.31.0 pyhd8ed1ab_0 conda-forge
scipy 1.7.3 py39hee8e79c_0 conda-forge
scp 0.14.1 pyhd8ed1ab_0 conda-forge
setuptools 59.5.0 pypi_0 pypi
six 1.16.0 pyh6c4a22f_0 conda-forge
snuggs 1.4.7 py_0 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
sqlite 3.37.0 h9cd32fc_0 conda-forge
superlance 1.1.0.dev0 pypi_0 pypi
supervisor 4.3.0.dev0 pypi_0 pypi
tblib 1.7.0 pyhd8ed1ab_0 conda-forge
tiledb 2.5.2 hf3d3071_0 conda-forge
tk 8.6.11 h27826a3_1 conda-forge
toolz 0.11.2 pyhd8ed1ab_0 conda-forge
tornado 6.1 py39h3811e60_2 conda-forge
tqdm 4.62.3 pyhd8ed1ab_0 conda-forge
traitlets 5.1.1 pyhd8ed1ab_0 conda-forge
trollflow2 0.11.1 pypi_0 pypi
trollimage 1.17.0 pyhd8ed1ab_0 conda-forge
trollmoves 0.9.0 pypi_0 pypi
trollsift 0.3.5 pyh44b312d_0 conda-forge
typing_extensions 4.0.1 pyha770c72_0 conda-forge
tzcode 2021e h7f98852_0 conda-forge
tzdata 2021e he74cb21_0 conda-forge
urllib3 1.26.7 pyhd8ed1ab_0 conda-forge
watchdog 2.1.6 pypi_0 pypi
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
wheel 0.37.0 pyhd8ed1ab_1 conda-forge
wrapt 1.13.3 pypi_0 pypi
xarray 0.20.2 pyhd8ed1ab_0 conda-forge
xerces-c 3.2.3 h8ce2273_4 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.2 h7f98852_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
yarl 1.7.2 pypi_0 pypi
zarr 2.10.3 pyhd8ed1ab_0 conda-forge
zeromq 4.3.4 h9c3ff4c_1 conda-forge
zict 2.0.0 py_0 conda-forge
zipp 3.6.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.11 h36c2ea0_1013 conda-forge
zstd 1.5.0 ha95c52a_0 conda-forge

Additional context
I'm not experienced with pytroll, but let me know if I can perform further tests to track down the problem.
@pnuu
Copy link
Member

pnuu commented Dec 16, 2021

This is not a Satpy issue, but check that your filepattern in trollstalker.ini contains start_time field. And that the message received by Trollflow2 contains that information.

@jdaniel006
Copy link
Author

Thank you!

I have already checked both, even copying the config files to avoid strange typos in the filepattern. As I said, same input files, same config files, issue disappears when downgrading satpy to 0.31.0

@jdaniel006
Copy link
Author

jdaniel006 commented Dec 16, 2021 •
edited

For the records (and to not paste all the config files with irrelevant info), this is the filepattern in trolstalker.ini

filepattern = {path}{satid:4s}-{instr:4s}-MSG{product_level:2d}-{base_algorithm_version:4s}-NA-{start_time:%Y%m%d%H%M%S.%f}000Z-{order_id:s}.nat

And the message logged:

2021-12-03 19:59:39,216 DEBG 'trollflow2' stdout output:
[DEBUG: 2021-12-03 19:59:39 : Listener] New message received: pytroll://dataset/seviri_native dataset meteo@SLTEC-VM-PRE 2021-12-03T18:59:39.215271 v1.01 application/json {"satid": "MSG4", "instr": "SEVI", "product_level": 15, "base_algorithm_version": "0100", "nominal_time": "2021-12-02T10:12:42.230000", "order_id": "NA", "start_time": "2021-12-02T10:12:42.230000", "dataset": [{"uri": "/home/meteo/METEOSAT/temp/MSG4-SEVI-MSG15-0100-NA-20211202101242.230000000Z-NA.nat", "uid": "MSG4-SEVI-MSG15-0100-NA-20211202101242.230000000Z-NA.nat"}], "sensor": ["seviri"]}

@pnuu
Copy link
Member

pnuu commented Dec 16, 2021

Oh, indeed! Sorry about that, my brain took an unnecessary shortcut.

And I can confirm, the Scene attributes are now completely empty.

@jdaniel006
Copy link
Author

No problem! Imagine my confusion after doing supposedly the same and getting different results.

@pnuu
Copy link
Member

pnuu commented Dec 16, 2021

It seems other plugins that use start_time update the Scene attributes with the message metadata. But I can't find any closed pull requests after 0.31.0 to point out why the attrs are now empty. Maybe that's hidden in some other PR? @djhoese?

@pnuu pnuu mentioned this issue Dec 16, 2021
Merged
4 tasks
@pnuu
Copy link
Member

pnuu commented Dec 16, 2021

@jdaniel006 the PR I made to Trollflow2 (linked above) should fix this.

@jdaniel006
Copy link
Author

Brilliant and quick!

Thanks a lot. I will test it in my development environment as soon as I can.

@pnuu
Copy link
Member

pnuu commented Dec 17, 2021

The fix in is now merged to main branch in Trollflow2, so upgrading should be a bit easier than using my branch. Not sure when we get round to creating a new release (I've started my holidays now).

@jdaniel006
Copy link
Author

jdaniel006 commented Dec 20, 2021 via email

@mraspaud mraspaud self-assigned this Dec 20, 2021
@mraspaud mraspaud added the bug label Dec 20, 2021
@mraspaud
Copy link
Member

A new version of trollflow2 has just been released: https://github.com/pytroll/trollflow2/releases/tag/v0.12.0
It should be available on pypi, and fixes this issue.

@jdaniel006
Copy link
Author

Upgraded with pip and working as expected.

Many thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mraspaud mraspaud

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mraspaud @pnuu @jdaniel006