Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hessian Reap/Sow Broken on Python3 Compiled Psi4 #833

JonathonMisiewicz opened this issue Oct 30, 2017 · 3 comments


Copy link

commented Oct 30, 2017

Reaping and sowing seem to be broken for Hessians when using Psi4 compiled with Python 3. None of the following errors could be reproduced when using Psi4 compiled with Python 2.

The following input file produces the following error:

molecule {
    H 1 R 
    H 1 R 2 A 
    H 1 R 2 A 3 D 

    R = 1.0043
    A = 120 
    D = 180 

set {
basis sto-3g

frequencies('ccsd(t)', dertype='gradient', mode='sow')
  File "/opt/vulcan/opt/vulcan/linux-x86_64/intel-16.0.1/psi4-master-avducsgnonqrcpftiesq432gagal6y7r/bin/psi4", line 260, in <module>
  File "<string>", line 33, in <module>
  File "/opt/vulcan/opt/vulcan/linux-x86_64/intel-16.0.1/psi4-master-avducsgnonqrcpftiesq432gagal6y7r/lib/psi4/driver/", line 1681, in frequency
    H, wfn = hessian(lowername, return_wfn=True, molecule=molecule, **kwargs)
  File "/opt/vulcan/opt/vulcan/linux-x86_64/intel-16.0.1/psi4-master-avducsgnonqrcpftiesq432gagal6y7r/lib/psi4/driver/", line 1332, in hessian 
    fmaster.write(p4util.format_options_for_input(moleculeclone, **kwargs))

TypeError: a bytes-like object is required, not 'str'

The culprit is exactly what the traceback says: fmaster is opened in binary mode and format_options_for_input returns a string. Python 2 would let you write a string to a binary file, but Python 3 expects bytes. I can fix this locally by adding the required .encode('utf-8') calls to that line as well as several lines afterwards, starting around 1370.

If that was all, I would just fix this myself, but sowing is useless without reaping, and the reaping bug is trickier. The displacement files include bytes encoded from UTF-8 and also bytes dumped from pickle (the gradient kwargs). In general, pickle bytes will not be intelligible in any text encoding scheme. Python 2's naive approach to bytes means this is not a problem. When you try to run one of those files with Python 3-compiled Psi4, however...

  File "/Users/jonathonmisiewicz/psi4/objdir/stage/usr/local/psi4/bin/psi4", line 230, in <module>
    content =
  File "/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 440: invalid start byte

Python 3 tries to read all the bytes as UTF-8. This fails for the pickled bytes, so Psi4 crashes. Even if you read the file as binary, you would still have to (somehow) instruct Python 3 which parts of the file should be UTF-8 decoded and which parts should be pickle-loaded.


This comment has been minimized.

Copy link

commented Oct 30, 2017

Yes, the key barrier is that sow/reap pickles its major pieces (mol, opts, call) for transport across p4 instances, and the py Molecule class has serious opposition to being pickled (side effect of the nice h2o.R = 1.5 capability). That's in general, and DGAS and I had a go a any sort of hack for py3 with no luck. The result is as you see all the sow/reap tests are deactivated in py3. The whole sow/reap scheme just needs reimplementation with json I/O and the option to Dask/Fireworks/etc. drive the freq.


This comment has been minimized.

Copy link

commented Jan 10, 2018

Update: fixed some of the py Molecule class issues (no more infinite recursion, yay!). Also gave the sow/reap pickles a serious stab, but was bested by unicode. So this issue will persist.


This comment has been minimized.

Copy link

commented Apr 13, 2018

This is a known issue and will stay broken until a fairly deep graph-based overhaul of the sow/reap code via QC JSON is undertaken. ETA Psi 1.3.

@dgasmith dgasmith closed this Apr 13, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
3 participants
You can’t perform that action at this time.