Cannot pickle local object

[sophiakrix]

Hi there!

I was trying to work with the obgl-biokg and execute the script at https://github.com/snap-stanford/ogb/blob/master/examples/linkproppred/biokg/run.py .

I was getting the following error:

Exception has occurred: AttributeError
Can't pickle local object 'main.<locals>.<lambda>'
  File "/.../src/ogb_original/dataloader.py", line 121, in one_shot_iterator
    for data in dataloader:
  File "/.../src/ogb_original/dataloader.py", line 112, in __next__
    data = next(self.iterator_tail)
  File "/.../src/ogb_original/model.py", line 238, in train_step
    positive_sample, negative_sample, subsampling_weight, mode = next(train_iterator)
  File "/...3/src/ogblbiokg/linkproppred_biokg.py", line 306, in main
    log = kge_model.train_step(kge_model, optimizer, train_iterator, args)
  File "/.../src/ogblbiokg/linkproppred_biokg.py", line 371, in <module>
    main(parse_args(params))

Could you help me how to handle this issue?

[weihua916]

I am not sure. Could you elaborate?
@hyren Do you have any idea for this?

[sophiakrix]

I literally just started the script, giving one fix parameter since it was required to specify train, validation or test set:

if __name__ == '__main__':
    
    params = ['--do_train']
    main(parse_args(params))

[sophiakrix]

Or could you maybe tell me how to run the examples/linkproppred/biokg/run.py script exactly? Maybe there is something I missed.

[weihua916]

Did you try this script?

[weihua916]

I am wondering what if you try Python 3.5. If this works for you, we may need to update the code to make it compatible with the newer Python.

I just found this error about pickle, and it seems to be related to the Python version..

[sophiakrix]

Hi @weihua916 ! Thanks for the response. Indeed, I think the python version is causing the error.
When I try with python 3.5, and want to install the ogb package, I get the following error:

(env_ogbl3.5) krixs@krixs-9TML87 ogb % pip install -e .         
DEPRECATION: Python 3.5 reached the end of its life on September 13th, 2020. Please upgrade your Python as Python 3.5 is no longer maintained. pip 21.0 will drop support for Python 3.5 in January 2021. pip 21.0 will remove support for this functionality.
Obtaining file:///Users/krixs/git/ogb
    ERROR: Command errored out with exit status 1:
     command: /opt/anaconda3/envs/env_ogbl3.5/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/krixs/git/ogb/setup.py'"'"'; __file__='"'"'/Users/krixs/git/ogb/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/_j/7l_9dpt555s_3r8x8xf9zr540000gn/T/pip-pip-egg-info-3g_h789o
         cwd: /Users/krixs/git/ogb/
    Complete output (8 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/Users/krixs/git/ogb/setup.py", line 9, in <module>
        from version import __version__
      File "/Users/krixs/git/ogb/ogb/version.py", line 19
        f'The OGB package is out of date. Your version is '
                                                          ^
    SyntaxError: invalid syntax
    ----------------------------------------
WARNING: Discarding file:///Users/krixs/git/ogb. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Could you tell me why the obg package (which I cloned from git via https just now) is out of date? Since I want to run the script run.py in the examples folder, I would need to clone the git repository and cannot use pip install ogb since this does not install the examples folder.

Therefore I thought I'd try with python 3.6. When I created a new environment with python3.6, installing the ogb packed worked. Yet, I get the following error when running the script:

  File "run.py", line 105
    **save_variable_list,
     ^
SyntaxError: invalid syntax

Which means it at least doesn't encounter the previous issue.

[rusty1s]

AFAIK, Python 3.6 is required since we make use of f-string syntax by now, while I think the BioKG example was created with Python 3.5. Can you try to remove the ** in **save_variable_list and see if this fixes your issues? This seems to be invalid syntax in Python >= 3.6.

[sophiakrix]

@rusty1s Thanks for the hint! Actually, without the issue with ** in **save_variable_list , the script is running. Yet just removing the ** is not possible, since it would be invalid syntax:

Traceback (most recent call last):
  File "/opt/anaconda3/envs/env_ogbl3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/anaconda3/envs/env_ogbl3.6/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/krixs/.vscode/extensions/ms-python.python-2021.9.1246542782/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/Users/krixs/.vscode/extensions/ms-python.python-2021.9.1246542782/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/Users/krixs/.vscode/extensions/ms-python.python-2021.9.1246542782/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/opt/anaconda3/envs/env_ogbl3.6/lib/python3.6/runpy.py", line 261, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
  File "/opt/anaconda3/envs/env_ogbl3.6/lib/python3.6/runpy.py", line 236, in _get_code_from_file
    code = compile(f.read(), fname, 'exec')
  File "/Users/krixs/git/ogb/examples/linkproppred/biokg/run.py", line 106
    'model_state_dict': model.state_dict(),
                      ^
SyntaxError: invalid syntax

By commenting out this line, the script runs. But this is of course not the way to go :D Do you have an idea how to write it differently?

[rusty1s]

I see. The correct code would be:

save_dict = {
        'model_state_dict': model.state_dict(),
        'optimizer_state_dict': optimizer.state_dict()},
}
save_dict.update(save_variable_list)
torch.save(save_dict, os.path.join(args.save_path, 'checkpoint'))

Does that work for you?

[sophiakrix]

@rusty1s , yes that made it work. Thanks a lot!

[weihua916]

Thank you two for figuring this out. @sophiakrix could you please create a pull request that works on Python 3.6 or newer? Thanks.