-
Notifications
You must be signed in to change notification settings - Fork 46
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CMIP(6) exclude_dirs #62
Comments
@aaronspring, thank you for pointing this out. This is something missing from the CMIP implementation. The |
@andersy005, are there further opportunities for consolidation/code reuse across CMIP and CESM? |
|
I think we should consider generalizing |
Sounds good. If you are not busy this coming week, let's discuss more about what these changes will look like |
With #63 You will be able to add name: cmip5_test_collection
collection_type: cmip5
data_sources:
root_dir:
name: GLADE
loc_type: posix
direct_access: True
urlpath: ./tests/sample_data/cmip/cmip5
exclude_dirs: ['files', 'latest'] |
Re-installed. YAML file: (I think this should work) name: cmip6_collection_mistral
collection_type: cmip6
data_sources:
root_dir:
name: MISTRAL
loc_type: posix
direct_access: True
urlpath: /work/kd0956/CMIP6
exclude_dirs: ['test_data/CMIP6','dc_repli/*','dc_test_data/*'] col = intake.open_esm_metadatastore(collection_input_definition=collection_input_definition, overwrite_existing=True)
Getting list of directories
---------------------------------------------------------------------------
PermissionError Traceback (most recent call last)
<ipython-input-15-554feb33ee30> in <module>
----> 1 col = intake.open_esm_metadatastore(collection_input_definition=collection_input_definition, overwrite_existing=True)
/work/mh0727/m300524/anaconda3/envs/xr/lib/python3.7/site-packages/intake_esm-2019.4.26.1.post15-py3.7.egg/intake_esm/core.py in __init__(self, collection_input_definition, collection_name, overwrite_existing, metadata)
64 collection_input_definition
65 )
---> 66 self.build_collection(overwrite_existing)
67
68 else:
/work/mh0727/m300524/anaconda3/envs/xr/lib/python3.7/site-packages/intake_esm-2019.4.26.1.post15-py3.7.egg/intake_esm/core.py in build_collection(self, overwrite_existing)
97 cc = ESMMetadataStoreCatalog.collection_types[ctype]
98 cc = cc(self.input_collection)
---> 99 cc.build()
100 self.get_built_collections()
101 self.open_collection(name)
/work/mh0727/m300524/anaconda3/envs/xr/lib/python3.7/site-packages/intake_esm-2019.4.26.1.post15-py3.7.egg/intake_esm/cmip.py in build(self)
145 Reference: CMIP6 DRS: http://goo.gl/v1drZl
146 """
--> 147 self.build_cmip(depth=9)
148
149 def _get_entry(self, directory):
/work/mh0727/m300524/anaconda3/envs/xr/lib/python3.7/site-packages/intake_esm-2019.4.26.1.post15-py3.7.egg/intake_esm/common.py in build_cmip(self, depth)
102 if not os.path.exists(self.root_dir):
103 raise NotADirectoryError(f'{os.path.abspath(self.root_dir)} does not exist')
--> 104 dirs = self.get_directories(root_dir=self.root_dir, depth=depth)
105 dfs = [self._parse_directory(directory, self.columns) for directory in dirs]
106 df = dd.from_delayed(dfs).compute()
/work/mh0727/m300524/anaconda3/envs/xr/lib/python3.7/site-packages/intake_esm-2019.4.26.1.post15-py3.7.egg/intake_esm/common.py in get_directories(self, root_dir, depth)
80
81 print('Getting list of directories')
---> 82 y = [x[0] for x in self.walk(root_dir, depth)]
83 diff = depth - 1
84 base = len(root_dir.split('/'))
/work/mh0727/m300524/anaconda3/envs/xr/lib/python3.7/site-packages/intake_esm-2019.4.26.1.post15-py3.7.egg/intake_esm/common.py in <listcomp>(.0)
80
81 print('Getting list of directories')
---> 82 y = [x[0] for x in self.walk(root_dir, depth)]
83 diff = depth - 1
84 base = len(root_dir.split('/'))
/work/mh0727/m300524/anaconda3/envs/xr/lib/python3.7/site-packages/intake_esm-2019.4.26.1.post15-py3.7.egg/intake_esm/common.py in walk(self, top, maxdepth)
69 if maxdepth > 1:
70 for path in dirs:
---> 71 for x in self.walk(path, maxdepth - 1):
72 yield x
73
/work/mh0727/m300524/anaconda3/envs/xr/lib/python3.7/site-packages/intake_esm-2019.4.26.1.post15-py3.7.egg/intake_esm/common.py in walk(self, top, maxdepth)
69 if maxdepth > 1:
70 for path in dirs:
---> 71 for x in self.walk(path, maxdepth - 1):
72 yield x
73
/work/mh0727/m300524/anaconda3/envs/xr/lib/python3.7/site-packages/intake_esm-2019.4.26.1.post15-py3.7.egg/intake_esm/common.py in walk(self, top, maxdepth)
64
65 dirs, nondirs = [], []
---> 66 for entry in os.scandir(top):
67 (dirs if entry.is_dir() else nondirs).append(entry.path)
68 yield top, dirs, nondirs
PermissionError: [Errno 13] Permission denied: '/work/kd0956/CMIP6/test_data/CMIP6' The error comes form
|
Looked into the code and dont see the problem.
|
I’m wondering why we can’t just wrap the directory scanning in a try block
and only include accessible files in the catalog.
|
When you get time, can you confirm that #64 works?
|
Closing this as it's been fixed in #64 |
Great job for the CMIP class!
Is it possible to add exclude_dirs for CMIP catalogs? The base directory contains other folders
which I don't have access too and therefore catalog building fails.
The text was updated successfully, but these errors were encountered: