pymake stalling on windows

[langevin-usgs]

@mwtoews, I think some of the recent changes you made for capturing the compile messages is causing a problem on windows. Our builds are stalling with SEAWAT and MF2000, for example. I can run compile.bat just fine from a windows terminal, but python seems to get hung up. We've verified that it happens on a couple of different windows machines. I'm guessing the issue is somewhere in here:

        proc = Popen([batchfile, ], stdout=PIPE, stderr=PIPE)
        while True:
            line = proc.stdout.readline()
            c = line.decode('utf-8')
            if c != '':
                c = c.rstrip('\r\n')
                print('{}'.format(c))
            else:
                break

Could you take a look?

[jdhughes-usgs]

You could debug using the make_mf2000.py and make_swtv4.py examples.

[mwtoews]

Is this with ifort or gfortran? (That except is from compile_with_ifort). I can see already that stderr=PIPE here should have been kept as stderr=STDOUT (changed in #28). But the other places where Popen is used should be fine with both stdout and stderr as PIPE as it receives these streams through proc.communicate().

[jdhughes-usgs]

ifort on Windows

[mwtoews]

I have ifort/windows, and I can't fully see this exact issue. Running buildall.bat within examples shows a different error:

D:\src\pymake\examples>buildall.bat

D:\src\pymake\examples>python buildall.py --appdir win32 --ifort --cl --ia32
...
D:\src\pymake\examples\tempbin>call "C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.8.275\windows\bin\compilervars.bat" ia32
Copyright (C) 1985-2017 Intel Corporation. All rights reserved.
Intel(R) Compiler 17.0 Update 8 (package 275)

**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.9.8
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x86'
vs2dt3_3.f
ifort: error #10142: directory '.\obj_temp\' does not exist
ifort: command line error: File not found:  '.\obj_temp\*.obj'

Cleaning up temporary source, object, and module files...
...

I'm not sure what's going on above, and this is the same behavior before any recent changes (I checked).

---

A simple "undo" of some of the change from #28 is (aka ff56ddb):

--- a/pymake/pymake.py
+++ b/pymake/pymake.py
@@ -21,7 +21,7 @@ import os
 import sys
 import traceback
 import shutil
-from subprocess import Popen, PIPE
+from subprocess import Popen, PIPE, STDOUT
 import argparse
 import datetime

@@ -803,7 +803,7 @@ def compile_with_ifort(srcfiles, target, fc, cc, objdir_temp, moddir_temp,
         makebatch(batchfile, fc, cc, fflags, cflags, srcfiles, target,
                   arch, objdir_temp, moddir_temp)
         #subprocess.check_call([batchfile, ])
-        proc = Popen([batchfile, ], stdout=PIPE, stderr=PIPE)
+        proc = Popen([batchfile, ], stdout=PIPE, stderr=STDOUT)
         while True:
             line = proc.stdout.readline()
             c = line.decode('utf-8')

this doesn't work for me (due the above older issue), but does it help anyone else?

[langevin-usgs]

This is what happens to me when I run buildall.bat on a windows 10 machine. It starts running just fine, and then just stops and hangs with no error messages when it gets to pes1bas6.f. I'll start trouble shooting and see if I can locate the problem.

C:\Users\langevin\langevin\dev\pymake.git\examples>python buildall.py --appdir win32 --ifort --cl --ia32
pymake graphing capabilities not available.
mf2000 will be placed in the directory:
    "win32"

mf2000 will be built using single precision floats.
mf2000 will be built as a release application.
mf2000 fortran code will be built with ifort.
mf2000 c/c++ code will be built with cl.

mf2000 will be built for ia32 architecture.
Attempting to download the file:
    https://water.usgs.gov/nrp/gwsoftware/modflow2000/mf2k1_19_01.tar.gz
 download attempt: 1
   file size: 4,911,590 bytes
     downloaded   204,800 bytes of 4,911,590 bytes (   4.1697%)
     downloaded   409,600 bytes of 4,911,590 bytes (   8.3395%)
     downloaded   614,400 bytes of 4,911,590 bytes (  12.5092%)
     downloaded   819,200 bytes of 4,911,590 bytes (  16.6789%)
     downloaded 1,024,000 bytes of 4,911,590 bytes (  20.8486%)
     downloaded 1,228,800 bytes of 4,911,590 bytes (  25.0184%)
     downloaded 1,433,600 bytes of 4,911,590 bytes (  29.1881%)
     downloaded 1,638,400 bytes of 4,911,590 bytes (  33.3578%)
     downloaded 1,843,200 bytes of 4,911,590 bytes (  37.5276%)
     downloaded 2,048,000 bytes of 4,911,590 bytes (  41.6973%)
     downloaded 2,252,800 bytes of 4,911,590 bytes (  45.8670%)
     downloaded 2,457,600 bytes of 4,911,590 bytes (  50.0367%)
     downloaded 2,662,400 bytes of 4,911,590 bytes (  54.2065%)
     downloaded 2,867,200 bytes of 4,911,590 bytes (  58.3762%)
     downloaded 3,072,000 bytes of 4,911,590 bytes (  62.5459%)
     downloaded 3,276,800 bytes of 4,911,590 bytes (  66.7157%)
     downloaded 3,481,600 bytes of 4,911,590 bytes (  70.8854%)
     downloaded 3,686,400 bytes of 4,911,590 bytes (  75.0551%)
     downloaded 3,891,200 bytes of 4,911,590 bytes (  79.2249%)
     downloaded 4,096,000 bytes of 4,911,590 bytes (  83.3946%)
     downloaded 4,300,800 bytes of 4,911,590 bytes (  87.5643%)
     downloaded 4,505,600 bytes of 4,911,590 bytes (  91.7340%)
     downloaded 4,710,400 bytes of 4,911,590 bytes (  95.9038%)
     downloaded 4,911,590 bytes of 4,911,590 bytes ( 100.0000%)
Deleting the zipfile...
Done downloading and extracting...
replacing select source files for mf2000
Removing..."temp\mf2k.1_19\src\beale2k"
Removing..."temp\mf2k.1_19\src\hydprgm"
Removing..."temp\mf2k.1_19\src\mf96to2k"
Removing..."temp\mf2k.1_19\src\mfpto2k"
Removing..."temp\mf2k.1_19\src\resan2k"
Removing..."temp\mf2k.1_19\src\ycint2k"
compiling...win32\mf2000.exe

source files are in:
    temp\mf2k.1_19\src
executable name to be created:
    C:\Users\langevin\langevin\dev\pymake.git\examples\win32\mf2000.exe

C:\Users\langevin\langevin\dev\pymake.git\examples>call "C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2019.2.190\windows\bin\compilervars.bat" ia32
Copyright (C) 1985-2018 Intel Corporation. All rights reserved.
Intel(R) Compiler 19.0 Update 2 (package 190)

**********************************************************************
** Visual Studio 2017 Developer Command Prompt v15.7.4
** Copyright (c) 2017 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x86'
utl6.f
gwf1lpf1.f
memchk.f
gwf1drn6.f
mhc1.f90
gutsdaf.f
hufutl2.f
obs1adv2.f
gwf1bcf6.f
sen1str6.f
sor5.f
sen1hfb6.f
gwf1rch6.f
sen1riv6.f
obs1riv6.f
sen1drt1.f
pes1bas6.f

[langevin-usgs]

And if I then kill the buildall.bat script which is hanging and run the compile.bat script that was created, it continues building the mf2000.exe script just fine.

[langevin-usgs]

It works now by replacing the Popen call to use STDOUT as suggested by @mwtoews. Thanks, Mike. I'll commit the fix.

proc = Popen([batchfile, ], stdout=PIPE, stderr=STDOUT)

[langevin-usgs]

Think this one is taken care of for now.