Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

added new notebook #174

Merged
merged 3 commits into from
Mar 22, 2018
Merged

added new notebook #174

merged 3 commits into from
Mar 22, 2018

Conversation

rabernat
Copy link
Member

This adds a new example notebook to the pangeo.pydata.org examples directory, as discussed in #130.

@mrocklin
Copy link
Member

Cool. It looks like you're using a personal branch of gcsfs here by @asford . Is that something that we should push on upstream to merge? cc @martindurant

Also, do we want to clear notebooks of outputs before merging in? All users will get this notebook in their home environment and we won't easily be able to change it later without potentially removing some of their data, so we want to make sure it's fairly clean.

@rabernat
Copy link
Member Author

This is the wrong notebook. There should be nothing custom or weird here. And all my nice notes are missing.

@mrocklin
Copy link
Member

Nicer! It looks like it references a globe image that isn't included. Can we link to some externally hosted image for that?

@rabernat
Copy link
Member Author

This is how that image appears in markdown

![globe image](http://marine.copernicus.eu/documents/IMG/SEALEVEL_GLO_SLA_MAP_L4_REP_OBSERVATIONS_008_027.png)

It already is an externally hosted image.

@rabernat
Copy link
Member Author

I am getting a new error now when I re-run this notebook:


_call exception: No such object: pangeo-data/dataset-duacs-rep-global-merged-allsat-phy-l4-v3-alt/.zgroup/.zarray
Traceback (most recent call last):
  File "/home/jovyan/.local/lib/python3.6/site-packages/gcsfs/core.py", line 432, in _call
    validate_response(r, path)
  File "/home/jovyan/.local/lib/python3.6/site-packages/gcsfs/core.py", line 162, in validate_response
    raise HtmlError(error)
gcsfs.utils.HtmlError: No such object: pangeo-data/dataset-duacs-rep-global-merged-allsat-phy-l4-v3-alt/.zgroup/.zarray
---------------------------------------------------------------------------
HtmlError                                 Traceback (most recent call last)
<ipython-input-3-88468ecbc9c1> in <module>()
      1 gcsmap = gcsfs.mapping.GCSMap('pangeo-data/dataset-duacs-rep-global-merged-allsat-phy-l4-v3-alt')
----> 2 ds = xr.open_zarr(gcsmap)
      3 ds

/opt/conda/lib/python3.6/site-packages/xarray/backends/zarr.py in open_zarr(store, group, synchronizer, auto_chunk, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables)
    458                                       synchronizer=synchronizer,
    459                                       group=group)
--> 460     ds = maybe_decode_store(zarr_store)
    461 
    462     # auto chunking needs to be here and not in ZarrStore because variable

/opt/conda/lib/python3.6/site-packages/xarray/backends/zarr.py in maybe_decode_store(store, lock)
    446             store, mask_and_scale=mask_and_scale, decode_times=decode_times,
    447             concat_characters=concat_characters, decode_coords=decode_coords,
--> 448             drop_variables=drop_variables)
    449 
    450         # TODO: this is where we would apply caching

/opt/conda/lib/python3.6/site-packages/xarray/conventions.py in decode_cf(obj, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables)
    592         encoding = obj.encoding
    593     elif isinstance(obj, AbstractDataStore):
--> 594         vars, attrs = obj.load()
    595         extra_coords = set()
    596         file_obj = obj

/opt/conda/lib/python3.6/site-packages/xarray/backends/common.py in load(self)
    215         """
    216         variables = FrozenOrderedDict((_decode_variable_name(k), v)
--> 217                                       for k, v in self.get_variables().items())
    218         attributes = FrozenOrderedDict(self.get_attrs())
    219         return variables, attributes

/opt/conda/lib/python3.6/site-packages/xarray/backends/zarr.py in get_variables(self)
    294     def get_variables(self):
    295         return FrozenOrderedDict((k, self.open_store_variable(k, v))
--> 296                                  for k, v in self.ds.arrays())
    297 
    298     def get_attrs(self):

/opt/conda/lib/python3.6/site-packages/xarray/core/utils.py in FrozenOrderedDict(*args, **kwargs)
    306 
    307 def FrozenOrderedDict(*args, **kwargs):
--> 308     return Frozen(OrderedDict(*args, **kwargs))
    309 
    310 

/opt/conda/lib/python3.6/site-packages/xarray/backends/zarr.py in <genexpr>(.0)
    293 
    294     def get_variables(self):
--> 295         return FrozenOrderedDict((k, self.open_store_variable(k, v))
    296                                  for k, v in self.ds.arrays())
    297 

/opt/conda/lib/python3.6/site-packages/zarr/hierarchy.py in arrays(self)
    452         for key in sorted(listdir(self._store, self._path)):
    453             path = self._key_prefix + key
--> 454             if contains_array(self._store, path):
    455                 yield key, Array(self._store, path=path, read_only=self._read_only,
    456                                  chunk_store=self._chunk_store,

/opt/conda/lib/python3.6/site-packages/zarr/storage.py in contains_array(store, path)
     70     prefix = _path_to_prefix(path)
     71     key = prefix + array_meta_key
---> 72     return key in store
     73 
     74 

~/.local/lib/python3.6/site-packages/gcsfs/mapping.py in __contains__(self, key)
     86 
     87     def __contains__(self, key):
---> 88         return self.gcs.exists(self._key_to_str(key))
     89 
     90     def __len__(self):

<decorator-gen-140> in exists(self, path)

~/.local/lib/python3.6/site-packages/gcsfs/core.py in _tracemethod(f, self, *args, **kwargs)
     49         logger.log(logging.DEBUG - 1, tb_io.getvalue())
     50 
---> 51     return f(self, *args, **kwargs)
     52 
     53 

~/.local/lib/python3.6/site-packages/gcsfs/core.py in exists(self, path)
    766         try:
    767             if key:
--> 768                 return bool(self.info(path))
    769             else:
    770                 if bucket in self.buckets:

<decorator-gen-141> in info(self, path)

~/.local/lib/python3.6/site-packages/gcsfs/core.py in _tracemethod(f, self, *args, **kwargs)
     49         logger.log(logging.DEBUG - 1, tb_io.getvalue())
     50 
---> 51     return f(self, *args, **kwargs)
     52 
     53 

~/.local/lib/python3.6/site-packages/gcsfs/core.py in info(self, path)
    800 
    801         try:
--> 802             return self._get_object(path)
    803         except FileNotFoundError:
    804             logger.debug("info FileNotFound at path: %s", path)

<decorator-gen-127> in _get_object(self, path)

~/.local/lib/python3.6/site-packages/gcsfs/core.py in _tracemethod(f, self, *args, **kwargs)
     49         logger.log(logging.DEBUG - 1, tb_io.getvalue())
     50 
---> 51     return f(self, *args, **kwargs)
     52 
     53 

~/.local/lib/python3.6/site-packages/gcsfs/core.py in _get_object(self, path)
    490 
    491         result = self._process_object(bucket, self._call('get', 'b/{}/o/{}',
--> 492                                                          bucket, key))
    493 
    494         return result

<decorator-gen-126> in _call(self, method, path, *args, **kwargs)

~/.local/lib/python3.6/site-packages/gcsfs/core.py in _tracemethod(f, self, *args, **kwargs)
     49         logger.log(logging.DEBUG - 1, tb_io.getvalue())
     50 
---> 51     return f(self, *args, **kwargs)
     52 
     53 

~/.local/lib/python3.6/site-packages/gcsfs/core.py in _call(self, method, path, *args, **kwargs)
    439                     # retry
    440                     continue
--> 441                 raise e
    442         try:
    443             out = r.json()

~/.local/lib/python3.6/site-packages/gcsfs/core.py in _call(self, method, path, *args, **kwargs)
    430                 time.sleep(2**retry - 1)
    431                 r = meth(self.base + path, params=kwargs, json=json)
--> 432                 validate_response(r, path)
    433                 break
    434             except (HtmlError, RequestException, GoogleAuthError) as e:

~/.local/lib/python3.6/site-packages/gcsfs/core.py in validate_response(r, path)
    160             raise ValueError("Bad Request: %s\n%s" % (path, msg))
    161         elif error:
--> 162             raise HtmlError(error)
    163         else:
    164             raise RuntimeError(m)

HtmlError: No such object: pangeo-data/dataset-duacs-rep-global-merged-allsat-phy-l4-v3-alt/.zgroup/.zarray

I have never seen this before, and it was working yesterday. Has something about the configuration changed?

@mrocklin
Copy link
Member

mrocklin commented Mar 22, 2018 via email

@mrocklin
Copy link
Member

mrocklin commented Mar 22, 2018 via email

@mrocklin
Copy link
Member

mrocklin commented Mar 22, 2018 via email

@rabernat rabernat mentioned this pull request Mar 22, 2018
Closed
@mrocklin
Copy link
Member

OK to merge?

@rabernat
Copy link
Member Author

Presumably the zarr issue is not specific to this notebook. So yes, ok with me.

@mrocklin mrocklin merged commit 99b6a5a into pangeo-data:master Mar 22, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rabernat @mrocklin