Ray integration yaml serialization error #586
Comments
|
Issue-Label Bot is automatically applying the label Links: app homepage, dashboard and code for this bot. |
|
Thanks for letting me know about this! I'll go over this issue in our weekly meeting. |
|
Any update on this? I'm facing similar issues. |
|
Having the same bug just now - this is specific to multi agent training in rllib only. The issue here is that the config contains a python lambda function. Your parser should probably just skip it. The relevant part is: See the docs here: and search for The OP provided a link to fully standalone example - you just need to add wandb instrumentation to it: |
|
FYI: This should be fixed in next version of PyYAML 5.1. You guys are using an old version PyYAML==4.2b4. Seems like they have released PyYAML 5.3: But there were some issues with 5.1 being backward compatible. So yeah... |
|
This is kind of required to log from rllib for more advanced scenarios... |
|
I have another one similar sounding error. wandb, version 0.8.36 initializing like this, inside a KFold loop: ...
wandb.init(
project=rootname.replace('/', '-').lower(),
name='run_{}_fr{}_ufr{}_{}of{}_fr'.format(
exp_uuid, froze_cycles, unfroze_cycles, run_i, n_splits
),
config=config,
group='fr_'+exp_uuid,
save_code=True,
tags=[
'parent_{}'.format(run),
'sklearn.model_selection.StratifiedKFold',
str(mod).split(' ')[1]
],
job_type='eval'
)
...getting ---------------------------------------------------------------------------
RepresenterError Traceback (most recent call last)
<ipython-input-37-80069da565fb> in <module>
70 str(mod).split(' ')[1]
71 ],
---> 72 job_type='eval'
73 )
74 learn.fit_one_cycle(froze_cycles)
/opt/conda/lib/python3.6/site-packages/wandb/__init__.py in init(job_type, dir, config, project, entity, reinit, tags, group, allow_val_change, resume, force, tensorboard, sync_tensorboard, monitor_gym, name, notes, id, magic, anonymous, config_exclude_keys, config_include_keys, save_code)
1142 include_keys=config_include_keys,
1143 allow_val_change=allow_val_change,
-> 1144 as_defaults=not allow_val_change)
1145
1146 # Access history to ensure resumed is set when resuming
/opt/conda/lib/python3.6/site-packages/wandb/wandb_config.py in _update(self, params, allow_val_change, as_defaults, exclude_keys, include_keys)
309 continue
310 self._items[key] = val
--> 311 self.persist()
312
313 def update(self, params, allow_val_change=False, exclude_keys=None, include_keys=None):
/opt/conda/lib/python3.6/site-packages/wandb/wandb_config.py in persist(self)
214 return
215 with open(path, "w") as conf_file:
--> 216 conf_file.write(str(self))
217 if wandb.run and wandb.run._jupyter_agent:
218 wandb.run._jupyter_agent.start()
/opt/conda/lib/python3.6/site-packages/wandb/wandb_config.py in __str__(self)
350 if as_dict: # adding an empty dictionary here causes a parse error
351 s += b'\n\n' + yaml.dump(as_dict, Dumper=yaml.SafeDumper, default_flow_style=False,
--> 352 allow_unicode=True, encoding='utf-8')
353 return s.decode("utf-8")
354
/opt/conda/lib/python3.6/site-packages/yaml/__init__.py in dump(data, stream, Dumper, **kwds)
288 If stream is None, return the produced string instead.
289 """
--> 290 return dump_all([data], stream, Dumper=Dumper, **kwds)
291
292 def safe_dump_all(documents, stream=None, **kwds):
/opt/conda/lib/python3.6/site-packages/yaml/__init__.py in dump_all(documents, stream, Dumper, default_style, default_flow_style, canonical, indent, width, allow_unicode, line_break, encoding, explicit_start, explicit_end, version, tags, sort_keys)
276 dumper.open()
277 for data in documents:
--> 278 dumper.represent(data)
279 dumper.close()
280 finally:
/opt/conda/lib/python3.6/site-packages/yaml/representer.py in represent(self, data)
25
26 def represent(self, data):
---> 27 node = self.represent_data(data)
28 self.serialize(node)
29 self.represented_objects = {}
/opt/conda/lib/python3.6/site-packages/yaml/representer.py in represent_data(self, data)
46 data_types = type(data).__mro__
47 if data_types[0] in self.yaml_representers:
---> 48 node = self.yaml_representers[data_types[0]](self, data)
49 else:
50 for data_type in data_types:
/opt/conda/lib/python3.6/site-packages/yaml/representer.py in represent_dict(self, data)
205
206 def represent_dict(self, data):
--> 207 return self.represent_mapping('tag:yaml.org,2002:map', data)
208
209 def represent_set(self, data):
/opt/conda/lib/python3.6/site-packages/yaml/representer.py in represent_mapping(self, tag, mapping, flow_style)
116 for item_key, item_value in mapping:
117 node_key = self.represent_data(item_key)
--> 118 node_value = self.represent_data(item_value)
119 if not (isinstance(node_key, ScalarNode) and not node_key.style):
120 best_style = False
/opt/conda/lib/python3.6/site-packages/yaml/representer.py in represent_data(self, data)
46 data_types = type(data).__mro__
47 if data_types[0] in self.yaml_representers:
---> 48 node = self.yaml_representers[data_types[0]](self, data)
49 else:
50 for data_type in data_types:
/opt/conda/lib/python3.6/site-packages/yaml/representer.py in represent_dict(self, data)
205
206 def represent_dict(self, data):
--> 207 return self.represent_mapping('tag:yaml.org,2002:map', data)
208
209 def represent_set(self, data):
/opt/conda/lib/python3.6/site-packages/yaml/representer.py in represent_mapping(self, tag, mapping, flow_style)
116 for item_key, item_value in mapping:
117 node_key = self.represent_data(item_key)
--> 118 node_value = self.represent_data(item_value)
119 if not (isinstance(node_key, ScalarNode) and not node_key.style):
120 best_style = False
/opt/conda/lib/python3.6/site-packages/yaml/representer.py in represent_data(self, data)
56 node = self.yaml_multi_representers[None](self, data)
57 elif None in self.yaml_representers:
---> 58 node = self.yaml_representers[None](self, data)
59 else:
60 node = ScalarNode(None, str(data))
/opt/conda/lib/python3.6/site-packages/yaml/representer.py in represent_undefined(self, data)
229
230 def represent_undefined(self, data):
--> 231 raise RepresenterError("cannot represent an object", data)
232
233 SafeRepresenter.add_representer(type(None),
RepresenterError: ('cannot represent an object', <function resnet50 at 0x7f6c5b49aa60>)YAML: import yaml
yaml.__version__
# >> '5.3' |
|
I figured out the problem for me was that I was trying to add the model to the wandb.config attribute. config = {
'exp_id': exp_uuid,
'input_size': 224,
'n_splits': n_splits,
'exp_method': StratifiedKFold,
'parent_run': run,
'model': models.resnet50}So I converted it to a string from the function name and it worked. config = {
'exp_id': exp_uuid,
'input_size': 224,
'n_splits': n_splits,
'exp_method': StratifiedKFold,
'parent_run': run,
'model': str(models.resnet50).split(' ')[1]} |
|
I ran across this issue myself when trying to run My solution is a little tacky but it works. I edited Basically, I found the If anyone has a better way or any guidance on how to do this properly, and more generally, please let me know. I'd be happy to contribute to this repo if I can. |
|
Actually found a better way: This looks through the dictionary of |
|
Found related commit. It is 4 behind master though. So for anyone looking, until this is fully merged, my solution and the commit's solution should work. |
wandb --version && python --version && unameDescription
Tried to use the wandb Ray tune integration. Training fails when serializing config with following error:
yaml.representer.RepresenterError: cannot represent an object: <function main.. at 0x7fd0fc4bfbf8>
What I Did
Started from simple cartpole example provided by Ray: https://raw.githubusercontent.com/ray-project/ray/master/rllib/examples/multiagent_cartpole.py
Imported wandb
from wandb.ray import WandbLogger
Included wandb configuration and set it as logger
The text was updated successfully, but these errors were encountered: