Python3 #127

kain88-de · 2018-04-26T13:42:56Z

except for two unit tests everything passes. Most changes should be harmless except for the tool registry, could be I broke that. But changing a dict during iteration isn't allowed in python 3 anymore.

fixes #44

orbeckst · 2018-04-27T18:50:46Z

The Python 2.7 tests fail because of

six.long does not seem to be found
DataFrame does not have sort() (?!?)

The Python 3 tests have problems installing Gromacs. I am looking into it.

orbeckst

@kain88-de can you think of a replacement for six.long() – it does not exist in the installed six.

The DataFrame.sort() error has been fixed in develop so this should be ok with a rebase. The Gromacs installation issues were raised in Becksteinlab/conda-gromacs-5.1.2#9 and Becksteinlab/conda-gromacs-5.1.2#10 (for Gromacs 5.1.2) and Becksteinlab/conda-gromacs-4.6.7#3 and Becksteinlab/conda-gromacs-4.6.7#4 (for 4.6.7).

orbeckst · 2018-04-29T19:21:15Z

gromacs/cbook.py

@@ -974,7 +975,7 @@ def edit_txt(filename, substitutions, newname=None):
            else:
                logger.debug("Deleting line %r", line)

-    target.seek(0L)
+    target.seek(six.long(0))


six.long() does not exist in six

In [4]: six.long --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-4-daab08199c2e> in <module>() ----> 1 six.long AttributeError: 'module' object has no attribute 'long' In [5]: six.__version__ Out[5]: '1.11.0'

I don't think we actually need the longs... so I am removing them.

orbeckst · 2018-04-29T19:21:36Z

gromacs/cbook.py

@@ -1040,7 +1041,7 @@ def remove_molecules_from_topology(filename, **kwargs):
                continue  # remove by skipping
            target.write(line)
    if autogenerated and n_removed > 0:
-        target.seek(0L)
+        target.seek(six.long(0))


six.long() does not exist in six

orbeckst · 2018-04-29T19:22:22Z

gromacs/run.py

@@ -332,7 +333,7 @@ def check_mdrun_success(logfile):
    except IOError:
        return None
    try:
-        log.seek(-1024L, 2)
+        log.seek(six.long(-1024), 2)


six.long() does not exist in six

kain88-de · 2018-05-01T06:43:49Z

If it works without the longs I would not use them anymore. Looks like the challenge now is python3 for your gromacs packages. I'm not sure what you mean by the DataFrame.sort issue.

orbeckst · 2018-05-01T15:38:52Z

Some tests used DataFrame.sort(), which does not exist anymore. But I fixed that. As you said, the Gromacs conda packages are the issue. The Gromacs devs are interested in creating their own conda packages but until that happens I still have to do something. When we hacked together our Gromacs conda packages we did not know about conda-forge but we should probably have invested the time to learn about it---it would be super-convenient if conda-forge could build the packages.

…

-- Oliver Beckstein email: orbeckst@gmail.com

Am Apr 30, 2018 um 23:43 schrieb Max Linke ***@***.***>: If it works without the longs I would not use them anymore. Looks like the challenge now is python3 for your gromacs packages. I'm not sure what you mean by the DataFrame.sort issue. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or mute the thread.

kain88-de · 2018-05-01T18:33:43Z

Some tests used DataFrame.sort(), which does not exist anymore. But I fixed that.

oh yeah this was replaced by sort_values. I was looking for this last week myself.

about the gromacs packages. Are there some emails about this on the mailing list? We do have some good contacts conda-forge to help out with this.

orbeckst · 2018-05-01T23:15:29Z

about the gromacs packages. Are there some emails about this on the mailing list? We do have some good contacts conda-forge to help out with this.

No, that wasn't a public discussion. I can email you offline if you like.

kain88-de · 2018-05-02T19:49:44Z

you can email me offline thanks.

codecov-io · 2018-05-10T04:52:28Z

Codecov Report

Merging #127 into develop will increase coverage by 0.13%.
The diff coverage is 70.58%.

@@             Coverage Diff             @@
##           develop     #127      +/-   ##
===========================================
+ Coverage    48.14%   48.28%   +0.13%     
===========================================
  Files           24       24              
  Lines         4831     4836       +5     
  Branches       731      733       +2     
===========================================
+ Hits          2326     2335       +9     
+ Misses        2352     2341      -11     
- Partials       153      160       +7

Impacted Files	Coverage Δ
gromacs/qsub.py	`29.78% <0%> (ø)`	⬆️
gromacs/fileformats/itp.py	`30.86% <0%> (ø)`	⬆️
gromacs/collections.py	`27.27% <100%> (ø)`	⬆️
gromacs/fileformats/xvg.py	`29.52% <100%> (ø)`	⬆️
gromacs/environment.py	`70.31% <100%> (+0.47%)`	⬆️
gromacs/run.py	`67.93% <100%> (-0.95%)`	⬇️
gromacs/core.py	`57.93% <50%> (+3.19%)`	⬆️
gromacs/config.py	`65.18% <50%> (+0.02%)`	⬆️
gromacs/cbook.py	`22.03% <66.66%> (+0.22%)`	⬆️
gromacs/utilities.py	`53.2% <77.77%> (-0.18%)`	⬇️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ebe740e...8d6ad76. Read the comment docs.

kain88-de · 2018-05-10T07:47:52Z

yay so python3 is running for at least one gromacs version

kain88-de · 2018-05-10T07:53:07Z

Ok so with the new packages from bioconda the tests seem to run. There are still failures but there is progress.

orbeckst · 2018-05-10T09:35:01Z

@ianmkenney has the same problem with Gromacs 2018 on conda in PR #130.

kain88-de · 2018-05-10T09:49:40Z

most things are done now. Some string handling issues are left.

kain88-de · 2018-05-13T10:31:32Z

@orbeckst this now sets all failing jobs to be allowed to fail. The old tests still work. Would you like to merge this as is? The last issues are with string handling but that might be better handled by someone who has used GromacsWrapper more then I did (I only needed the xvg reader once)

orbeckst · 2018-05-16T01:00:44Z

I'd like to get the Python 3/Gromacs 4.6.5 tests to pass so that this PR really fixes #44.

I'll try to look at it over the next few days.

kain88-de · 2018-05-18T08:43:43Z

I'm down to three errors now. The last ones fail when gromacs is rerun for amber forcefield. So something internally is messed up. Fixing these errors is not a lot more difficult then pocking in the debugger and some light googling.

kain88-de · 2018-05-18T08:49:57Z

ok so yeah string handling fixes for python3 have broken python2 -.-

kain88-de · 2018-05-18T12:07:37Z

@ianmkenney do you know why the last three tests fail?

orbeckst · 2018-06-12T06:53:44Z

I have no idea why these Test<FF>.test_mdrun tests fail under 3.6 but pass under 2.7. Why are the numerical results different? However, this looks as if it could be an easy win if it's something obviously stupid. ... and then this PR can be merged and we can start fixing 2018.

kain88-de · 2018-06-12T07:47:41Z

I would need some help to investigate those errors. It would be helpful to know the exact gromacs commands that are executed for this test in python 2.7 and 3.6. Without knowing the library it's not straightforward to understand the program flow that is happening.

kain88-de · 2018-07-28T17:50:25Z

@ianmkenney did you have a chance to fixup the gromacs errors?

ianmkenney · 2018-07-30T16:26:34Z

@kain88-de I haven't had much time to look at this issue, I can take a look in the next couple of weeks hopefully

orbeckst · 2018-07-30T20:19:32Z

Given our sluggish pace of development on GW (and the general "wild west coding" attitude in this project ;-) ) I would go ahead and merge and immediately raise an issue for Python 3.x correctness. It would be nice to merge with everything passing (#127 (comment)) but at least it will allow us to move forward.

Any objections?

kain88-de · 2018-07-30T20:28:18Z

I don't mind if you want to merge this. Just keep in mind I don't have much time nor expertise to fix things in this project.

…

On Mon, Jul 30, 2018 at 10:19 PM Oliver Beckstein ***@***.***> wrote: Given our sluggish pace of development on GW (and the general "wild west coding" attitude in this project ;-) ) I would go ahead and merge and immediately raise an issue for Python 3.x correctness. It would be nice to merge with everything passing (#127 (comment) <#127 (comment)>) but at least it will allow us to move forward. Any objections? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#127 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEGnVv9MOUSMVZYELvZX_B3o57if-p_5ks5uL2pVgaJpZM4TlNNC> .

orbeckst · 2018-07-30T23:55:08Z

I totally understand.

there are still some Grompp errors I don't know about

Code used to have 0L and -1024L but this is not supported in Py3 and it does not seem to be necessary in Py2 as shown in the docs https://docs.python.org/2/library/stdtypes.html#file.seek

- use binary mode reading and writing when necessary - replace map operator - explicit list - fix run - fix pickling - .... and more

orbeckst · 2018-07-31T01:02:39Z

I am rewriting the history to clean up the commits. I will force-push to your branch and then merge.

kain88-de force-pushed the python3 branch from 870779a to d71d149 Compare April 26, 2018 13:44

orbeckst self-assigned this Apr 27, 2018

orbeckst requested changes Apr 29, 2018

View reviewed changes

orbeckst mentioned this pull request Apr 30, 2018

WIP: Issue #44 Python 3 #101

Closed

orbeckst mentioned this pull request May 8, 2018

use Bioconda gromacs packages for testing #129

Closed

3 tasks

kain88-de force-pushed the python3 branch from 7367ca0 to 3d513ce Compare May 10, 2018 08:14

orbeckst mentioned this pull request May 16, 2018

Changed gromacs package for testing #130

Closed

ianmkenney mentioned this pull request May 16, 2018

Gromacs 2018 and pytest low-performance custom plugin to control -nt #102

Merged

orbeckst mentioned this pull request May 16, 2018

test Gromacs 2018 #96

Closed

3 tasks

kain88-de force-pushed the python3 branch 2 times, most recently from d2d8693 to 9d20ea3 Compare May 18, 2018 09:06

orbeckst mentioned this pull request Jun 12, 2018

python3 corrections #131

Closed

orbeckst approved these changes Jul 31, 2018

View reviewed changes

kain88-de and others added 13 commits July 30, 2018 17:57

minimal changes for python 3

80b2153

there are still some Grompp errors I don't know about

update author/changes/readme

a9ee4be

update travis to test python3

d505134

more updates for python 3

24077a5

simpler approach with explicit copy

7b45056

use int arguments for file.seek()

00302b5

Code used to have 0L and -1024L but this is not supported in Py3 and it does not seem to be necessary in Py2 as shown in the docs https://docs.python.org/2/library/stdtypes.html#file.seek

use bioconda gromacs version

3871148

fix build issue

61cd09e

add conda-forge for 2018 version

1fe0c82

fix encoding

f1d8452

basestring fix

d4a2646

various Python 3 fixes

e97da07

- use binary mode reading and writing when necessary - replace map operator - explicit list - fix run - fix pickling - .... and more

python 2 fix

a9cd1c6

only allow gromacs2018 to fail in CI

8d6ad76

orbeckst force-pushed the python3 branch from 549a891 to 8d6ad76 Compare July 31, 2018 01:08

orbeckst merged commit f52d14c into Becksteinlab:develop Jul 31, 2018

kain88-de deleted the python3 branch July 31, 2018 07:51

This was referenced Aug 1, 2018

Test*.test_mdrun failures under Python 3.x #132

Closed

tools not loaded for Gromacs 2016/2018 in Python 3 on macOS #133

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python3 #127

Python3 #127

kain88-de commented Apr 26, 2018 •

edited

orbeckst commented Apr 27, 2018

orbeckst left a comment

orbeckst Apr 29, 2018

orbeckst Apr 29, 2018

orbeckst Apr 30, 2018

orbeckst Apr 29, 2018

orbeckst Apr 29, 2018

kain88-de commented May 1, 2018

orbeckst commented May 1, 2018 via email

kain88-de commented May 1, 2018

orbeckst commented May 1, 2018

kain88-de commented May 2, 2018

codecov-io commented May 10, 2018 •

edited

kain88-de commented May 10, 2018

kain88-de commented May 10, 2018

orbeckst commented May 10, 2018

kain88-de commented May 10, 2018

kain88-de commented May 13, 2018

orbeckst commented May 16, 2018

kain88-de commented May 18, 2018

kain88-de commented May 18, 2018

kain88-de commented May 18, 2018

orbeckst commented Jun 12, 2018 •

edited

kain88-de commented Jun 12, 2018

kain88-de commented Jul 28, 2018

ianmkenney commented Jul 30, 2018

orbeckst commented Jul 30, 2018

kain88-de commented Jul 30, 2018 via email

orbeckst commented Jul 30, 2018

orbeckst commented Jul 31, 2018

Python3 #127

Python3 #127

Conversation

kain88-de commented Apr 26, 2018 • edited

orbeckst commented Apr 27, 2018

orbeckst left a comment

Choose a reason for hiding this comment

orbeckst Apr 29, 2018

Choose a reason for hiding this comment

orbeckst Apr 29, 2018

Choose a reason for hiding this comment

orbeckst Apr 30, 2018

Choose a reason for hiding this comment

orbeckst Apr 29, 2018

Choose a reason for hiding this comment

orbeckst Apr 29, 2018

Choose a reason for hiding this comment

kain88-de commented May 1, 2018

orbeckst commented May 1, 2018 via email

kain88-de commented May 1, 2018

orbeckst commented May 1, 2018

kain88-de commented May 2, 2018

codecov-io commented May 10, 2018 • edited

Codecov Report

kain88-de commented May 10, 2018

kain88-de commented May 10, 2018

orbeckst commented May 10, 2018

kain88-de commented May 10, 2018

kain88-de commented May 13, 2018

orbeckst commented May 16, 2018

kain88-de commented May 18, 2018

kain88-de commented May 18, 2018

kain88-de commented May 18, 2018

orbeckst commented Jun 12, 2018 • edited

kain88-de commented Jun 12, 2018

kain88-de commented Jul 28, 2018

ianmkenney commented Jul 30, 2018

orbeckst commented Jul 30, 2018

kain88-de commented Jul 30, 2018 via email

orbeckst commented Jul 30, 2018

orbeckst commented Jul 31, 2018

kain88-de commented Apr 26, 2018 •

edited

codecov-io commented May 10, 2018 •

edited

orbeckst commented Jun 12, 2018 •

edited