Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

save_datasets does not save dependencies available due to unload=False #1138

Closed
gerritholl opened this issue Apr 7, 2020 · 7 comments · Fixed by #1864
Closed

save_datasets does not save dependencies available due to unload=False #1138

gerritholl opened this issue Apr 7, 2020 · 7 comments · Fixed by #1864

Comments

@gerritholl
Copy link
Collaborator

Describe the bug

When I load a composite with unload=False, then run sc.save_datasets, the datasets loaded as dependencies are not saved.

To Reproduce

import tempfile
from satpy.scene import Scene
from glob import glob

fn_sev = glob("/media/nas/x21308/scratch/SEVIRI/H-000*201904151000*")
sc = Scene(filenames={"seviri_l1b_hrit": fn_sev})

sc.load(["overview"], unload=False)

print("Number of unique dataset names:", len({d.name for d in sc.keys()}))
with tempfile.TemporaryDirectory() as td:
    sc.save_datasets(filename=td + "/test_{name:s}.tif")
    print("Number of files found:", len(glob(td + "/*.tif")))

Expected behavior

I expect:

Number of unique dataset names: 4
Number of files found: 4

Actual results

It's actually only writing out one file, so Number of files found: 1.

``` [DEBUG: 2020-04-07 19:06:32 : satpy.scene] Setting 'PPP_CONFIG_DIR' to '/home/gholl/checkouts/satpy/satpy/etc/' [181/1431] [DEBUG: 2020-04-07 19:06:32 : satpy.readers] Reading ['/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/etc/readers/seviri_l1b_hrit.yaml', '/home/gholl/checkouts/satpy/satpy/etc/readers/seviri_l1b_hrit.yaml'] [DEBUG: 2020-04-07 19:06:32 : satpy.readers.yaml_reader] Assigning to seviri_l1b_hrit: ['/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000007___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000012___-201904151000-__' , '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000003___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000023___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000018___-2019 04151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000016___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000001___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-00 0010___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000017___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000004___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________- HRV______-000002___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000014___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000013___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-M SG4________-HRV______-000021___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000019___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000020___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H- 000-MSG4__-MSG4________-HRV______-000022___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000024___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000015___-201904151000-__', '/media/nas/x21308/scrat ch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000006___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000008___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000009___-201904151000-__', '/media/nas/ x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000011___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-HRV______-000005___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_016___-000008___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_016___-000003___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_016___-000004___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_016___-000002___-201904 151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_016___-000005___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_016___-000007___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_016___-0000 01___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_016___-000006___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_039___-000001___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR _039___-000005___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_039___-000003___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_039___-000007___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG 4________-IR_039___-000004___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_039___-000006___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_039___-000008___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-00 0-MSG4__-MSG4________-IR_039___-000002___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_087___-000002___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_087___-000006___-201904151000-__', '/media/nas/x21308/scratch /SEVIRI/H-000-MSG4__-MSG4________-IR_087___-000004___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_087___-000001___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_087___-000003___-201904151000-__', '/media/nas/x2 1308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_087___-000005___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_087___-000008___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_087___-000007___-201904151000-__', '/ media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_097___-000004___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_097___-000005___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_097___-000007___-20190415 1000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_097___-000002___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_097___-000006___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_097___-000003 ___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_097___-000001___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_097___-000008___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_1 08___-000008___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_108___-000002___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_108___-000003___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4_ _______-IR_108___-000006___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_108___-000005___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_108___-000007___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000- MSG4__-MSG4________-IR_108___-000004___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_108___-000001___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_120___-000005___-201904151000-__', '/media/nas/x21308/scratch/S EVIRI/H-000-MSG4__-MSG4________-IR_120___-000001___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_120___-000007___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_120___-000008___-201904151000-__', '/media/nas/x213 08/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_120___-000004___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_120___-000003___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_120___-000002___-201904151000-__', '/me dia/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_120___-000006___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_134___-000004___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_134___-000001___-2019041510 00-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_134___-000006___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_134___-000003___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_134___-000005__ _-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_134___-000007___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_134___-000008___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-IR_134 ___-000002___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-VIS006___-000006___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-VIS006___-000007___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4___ _____-VIS006___-000005___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-VIS006___-000002___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-VIS006___-000008___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MS G4__-MSG4________-VIS006___-000004___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-VIS006___-000003___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-VIS006___-000001___-201904151000-__', '/media/nas/x21308/scratch/SEV IRI/H-000-MSG4__-MSG4________-VIS008___-000003___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-VIS008___-000002___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-VIS008___-000007___-201904151000-__', '/media/nas/x21308 /scratch/SEVIRI/H-000-MSG4__-MSG4________-VIS008___-000004___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-VIS008___-000005___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-VIS008___-000001___-201904151000-__', '/medi a/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-VIS008___-000008___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-VIS008___-000006___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-WV_062___-000004___-201904151000 -__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-WV_062___-000007___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-WV_062___-000001___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-WV_062___-000008___- 201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-WV_062___-000002___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-WV_062___-000006___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-WV_062__ _-000005___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-WV_062___-000003___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-WV_073___-000004___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4_____ ___-WV_073___-000007___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-WV_073___-000001___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-WV_073___-000008___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4 __-MSG4________-WV_073___-000002___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-WV_073___-000005___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-WV_073___-000003___-201904151000-__', '/media/nas/x21308/scratch/SEVIR I/H-000-MSG4__-MSG4________-WV_073___-000006___-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-_________-PRO______-201904151000-__', '/media/nas/x21308/scratch/SEVIRI/H-000-MSG4__-MSG4________-_________-EPI______-201904151000-__'] [INFO: 2020-04-07 19:06:32 : hrit_msg] No IMPF configuration field found in prologue. [DEBUG: 2020-04-07 19:06:32 : hrit_msg] Computing actual satellite position [DEBUG: 2020-04-07 19:06:33 : satpy.composites] Looking for composites config file seviri.yaml [DEBUG: 2020-04-07 19:06:33 : satpy.composites] Looking for composites config file visir.yaml /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.005994 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.006007 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.005722 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.005801 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.005931 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.005968 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.006117 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.005575 /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordin ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.009222 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.009216 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.008912 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.009284 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.008759 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.009125 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.008656 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.009161 /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordi$ ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordi$ ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordi$ ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordi$ ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordi$ ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordi$ ate-reference-systems proj_string = self.to_proj4() /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordi$ ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.006479 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.005621 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.005457 [DEBUG: 2020-04-07 19:06:33 : hrit_msg] Calibration time 0:00:00.006019 [DEBUG: 2020-04-07 19:06:34 : hrit_msg] Calibration time 0:00:00.005976 [DEBUG: 2020-04-07 19:06:34 : hrit_msg] Calibration time 0:00:00.005500 [DEBUG: 2020-04-07 19:06:34 : hrit_msg] Calibration time 0:00:00.005501 [DEBUG: 2020-04-07 19:06:34 : hrit_msg] Calibration time 0:00:00.005440 /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordi$ ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-04-07 19:06:34 : satpy.composites] Applying sun zen correction [DEBUG: 2020-04-07 19:06:34 : satpy.composites] Computing sun zenith angles. [DEBUG: 2020-04-07 19:06:34 : satpy.composites] Apply the standard sun-zenith correction [1/cos(sunz)] [DEBUG: 2020-04-07 19:06:34 : satpy.composites] Sun-zenith correction applied. Computation time: 0.0 (sec) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordi$ ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-04-07 19:06:34 : satpy.composites] Applying sun zen correction [DEBUG: 2020-04-07 19:06:34 : satpy.composites] Computing sun zenith angles. [DEBUG: 2020-04-07 19:06:34 : satpy.composites] Apply the standard sun-zenith correction [1/cos(sunz)] [DEBUG: 2020-04-07 19:06:34 : satpy.composites] Sun-zenith correction applied. Computation time: 0.0 (sec) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/pyproj/crs/crs.py:543: UserWarning: You will likely lose important projection information when converting to a PROJ string from another format. See: https://proj.org/faq.html#what-is-the-best-format-for-describing-coordi$ ate-reference-systems proj_string = self.to_proj4() [DEBUG: 2020-04-07 19:06:34 : satpy.writers] Reading ['/data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/satpy/etc/writers/geotiff.yaml', '/home/gholl/checkouts/satpy/satpy/etc/writers/geotiff.yaml'] [DEBUG: 2020-04-07 19:06:34 : satpy.writers] Enhancement configuration options: [{'name': 'inverse', 'method': , 'args': [[False, False, True]]}, {'name': 'stretch', 'method': , 'kwargs': {'stretch': 'linear'}}$ {'name': 'gamma', 'method': , 'kwargs': {'gamma': 1.6}}] [DEBUG: 2020-04-07 19:06:34 : trollimage.xrimage] Applying invert with parameters [False, False, True] [DEBUG: 2020-04-07 19:06:34 : trollimage.xrimage] Applying stretch linear with parameters {} [DEBUG: 2020-04-07 19:06:34 : trollimage.xrimage] Perform a linear contrast stretch. [DEBUG: 2020-04-07 19:06:34 : trollimage.xrimage] Calculate the histogram quantiles: [DEBUG: 2020-04-07 19:06:34 : trollimage.xrimage] Left and right quantiles: 0.005 0.005 [DEBUG: 2020-04-07 19:06:34 : trollimage.xrimage] Applying gamma 1.6 [DEBUG: 2020-04-07 19:06:34 : rasterio.env] Entering env context: [DEBUG: 2020-04-07 19:06:34 : rasterio.env] Starting outermost env [DEBUG: 2020-04-07 19:06:34 : rasterio.env] No GDAL environment exists [DEBUG: 2020-04-07 19:06:34 : rasterio.env] New GDAL environment created [DEBUG: 2020-04-07 19:06:34 : rasterio._env] GDAL_DATA found in environment: '/data/gholl/miniconda3/envs/py38/share/gdal'. [DEBUG: 2020-04-07 19:06:34 : rasterio._env] PROJ_LIB found in environment: '/data/gholl/miniconda3/envs/py38/share/proj'. [DEBUG: 2020-04-07 19:06:34 : rasterio._env] Started GDALEnv . [DEBUG: 2020-04-07 19:06:34 : rasterio.env] Entered env context: [DEBUG: 2020-04-07 19:06:34 : rasterio._io] Path: UnparsedPath(path='/tmp/tmpafr59wzi/test_overview.tif'), mode: w, driver: GTiff [DEBUG: 2020-04-07 19:06:34 : rasterio._io] Option: ('COMPRESS', b'DEFLATE') [DEBUG: 2020-04-07 19:06:34 : rasterio._io] Option: ('ZLEVEL', b'6') [DEBUG: 2020-04-07 19:06:34 : rasterio._io] Option: ('PHOTOMETRIC', b'RGB') [DEBUG: 2020-04-07 19:06:34 : rasterio._io] Skipped delete for overwrite. Dataset does not exist: /tmp/tmpafr59wzi/test_overview.tif [DEBUG: 2020-04-07 19:06:34 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000 [DEBUG: 2020-04-07 19:06:34 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000 [DEBUG: 2020-04-07 19:06:34 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000 [DEBUG: 2020-04-07 19:06:34 : rasterio._base] Nodata success: 0, Nodata value: -10000000000.000000 [DEBUG: 2020-04-07 19:06:34 : rasterio.env] Exiting env context: [DEBUG: 2020-04-07 19:06:34 : rasterio.env] Cleared existing options [DEBUG: 2020-04-07 19:06:34 : rasterio._env] Stopped GDALEnv . [DEBUG: 2020-04-07 19:06:34 : rasterio.env] Exiting outermost env [DEBUG: 2020-04-07 19:06:34 : rasterio.env] Exited env context: [INFO: 2020-04-07 19:06:34 : satpy.writers] Computing and writing results... /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in cos return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in sin return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in greater_equal return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in greater return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in log return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in cos return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in sin return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in greater_equal return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in greater return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in greater_equal return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in cos return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in log return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in sin return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in greater return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in greater return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in log return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in greater return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in log return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in cos return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in sin return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in greater_equal return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in greater return func(*(_execute_task(a, cache) for a in args)) /data/gholl/miniconda3/envs/py38/lib/python3.8/site-packages/dask/core.py:121: RuntimeWarning: invalid value encountered in log return func(*(_execute_task(a, cache) for a in args)) [DEBUG: 2020-04-07 19:06:36 : trollimage.xrimage] Interval: left= array([ 0. , 0. , -322.64242554]) Coordinates: quantile float64 0.005 Dimensions without coordinates: bands, right= array([ 102.68503157, 119.72653831, -213.19895935]) Coordinates: quantile float64 0.995 Dimensions without coordinates: bands [DEBUG: 2020-04-07 19:06:38 : rasterio._io] Dataset has been stopped. Number of unique dataset names: 4 Number of files found: 1 ```

Environment Info:

  • OS: openSUSE 15
  • Satpy Version: 0.21.0
  • PyResample Version: 1.15.0+2.g3d6d38d

Additional context
There are also a lot of warnings, but I don't think they are related to this problem.

@djhoese
Copy link
Member

djhoese commented Apr 7, 2020

Just for reference, here are the DatasetIDs in the Scene:

In [4]: scn.keys()
Out[4]:
[DatasetID(name='IR_108', wavelength=(9.8, 10.8, 11.8), resolution=3000.403165817, polarization=None, calibration='brightness_temperature', level=None, modifiers=()),
 DatasetID(name='VIS006', wavelength=(0.56, 0.635, 0.71), resolution=3000.403165817, polarization=None, calibration='reflectance', level=None, modifiers=()),
 DatasetID(name='VIS006', wavelength=(0.56, 0.635, 0.71), resolution=3000.403165817, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',)),
 DatasetID(name='VIS008', wavelength=(0.74, 0.81, 0.88), resolution=3000.403165817, polarization=None, calibration='reflectance', level=None, modifiers=()),
 DatasetID(name='VIS008', wavelength=(0.74, 0.81, 0.88), resolution=3000.403165817, polarization=None, calibration='reflectance', level=None, modifiers=('sunz_corrected',)),
 DatasetID(name='overview', wavelength=None, resolution=3000.403165817, polarization=None, calibration=None, level=None, modifiers=None)]

@djhoese
Copy link
Member

djhoese commented Apr 7, 2020

Oh! I think this is the code that matters:

satpy/satpy/scene.py

Lines 1344 to 1345 in 446af3a

datasets = [self.datasets.get(ds) for ds in self.wishlist]
datasets = [ds for ds in datasets if ds is not None]

The Scene is only going to save the datasets you explicitly requested (the ones on your wishlist). Since in your case everything except overview was a dependency of something you did ask for, it doesn't get saved.

@djhoese
Copy link
Member

djhoese commented Apr 7, 2020

Additionally, if you'd like to save these, you can specify them in the datasets keyword argument to save_datasets.

@mraspaud
Copy link
Member

mraspaud commented Apr 7, 2020

This is a feature not a bug. Only requested datasets will be saved.

@djhoese
Copy link
Member

djhoese commented Apr 7, 2020

Sorry, one more: This is intended. Most users consider it undesired to have the Scene save things they didn't ask for.

Edit: Of course, Martin gets to it first.

@gerritholl
Copy link
Collaborator Author

gerritholl commented Apr 8, 2020

@mraspaud The documentation says otherwise:

Save all the datasets present in a scene to disk using writer.

Also, if a user didn't ask to keep the dependencies, why would they pass unload=False? The reason I'm passing unload=False is because I want to store all the dependencies (for debugging purposes). Doesn't explicitly passing unload=False means the user did ask for them?

@mraspaud
Copy link
Member

mraspaud commented Apr 8, 2020

Fair enough. I see a point in your use case, so PR is welcome :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants