No Timeout when trying to to execute on the 32-bit server from 64bit client

[learningmadeeasyfun]

I faced an issue that despite the fact that I set timeout for invocation of 32-bit server from 64bit client the timeout was not happening. The issue was found in below file and code:

msl\loadlib\client64.py :

        # start the 32-bit server
        self._proc = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
        try:
            utils.wait_for_server(host, port, timeout)
        except ConnectionTimeoutError as err:
            self._proc.wait() #  line 201

As per the above code even though there is a connection timeout exception occurs the code waits for INFINITY time for the subprocess to be killed as per below code:

lib\subprocess.py:

       def wait(self, timeout=None, endtime=None):  
  ................................................................................
            if timeout is None:
                timeout_millis = _winapi.INFINITE
            else:
                timeout_millis = int(timeout * 1000)
            if self.returncode is None:
                result = _winapi.WaitForSingleObject(self._handle,
                                                    timeout_millis)
                if result == _winapi.WAIT_TIMEOUT:
                    raise TimeoutExpired(self.args, timeout)
                self.returncode = _winapi.GetExitCodeProcess(self._handle)
            return self.returncode

Hence the code waits for infinity time period - I have made a fix for same it works but not sure if it is the correct one:

Just change line 201 in msl\loadlib\client64.py from self._proc.wait() to **self._proc.wait(timeout=timeout)**

[jborbely]

Your suggestion to modify line 201 sounds reasonable. I just have a few comments/questions:

1. Does the infinite hanging occur every time you run your code or does it only happen occasionally?
2. If self._proc.wait(timeout=timeout) is used then instantiating Client64 would take 2 x timeout seconds to finish for the issue you experienced. Not sure if some other value, e.g., self._proc.wait(timeout=1) should be used instead.
3. The timeout argument was added to Popen.wait in Python 3.3. This package is still continuing to support Python 2.7. Do you have a suggestion for a Python 2.7 fix?
4. If Popen.wait times out then a TimeoutExpired exception is raised -- which should be caught and ignored. Does this exception get raised with your fix instead of the custom msl.loadlib.ConnectionTimeoutError?

I am curious to see how this issue occurred in your code. Could you share the code that gets executed by the 32-bit server? I'd like to write a test case for this issue.

[learningmadeeasyfun]

Thanks for your comments.

No the issue does not occurs frequently but specifically first couple of times when I unlock my system after locking with application running and re-run the app. The code is the sample code available at : Here with a little bit modification . I am using Python 3.6 and probably upgrade to 3.9 shortly.

I have changed my code to self._proc.wait(timeout=1)

C:\Users\python>python
Python 3.6.2 (v3.6.2:5fd33b5, Jul  8 2017, 04:57:36) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from my_client import MyClient
>>> c = MyClient()

==== NEED TO PRESS CTRL + C ===============

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\python\my_client", line 9, in __init__
    super(my_client, self).__init__(module32='my_server', timeout=10.0, rpc_timeout=10.0)
  File "C:\Python36\lib\site-packages\msl\loadlib\client64.py", line 199, in __init__
    utils.wait_for_server(host, port, timeout)
  File "C:\Python36\lib\site-packages\msl\loadlib\utils.py", line 283, in wait_for_server
    'Timeout after {:.1f} seconds. Could not connect to {}:{}'.format(timeout, host, port)
msl.loadlib.exceptions.ConnectionTimeoutError: Timeout after 10.0 seconds. Could not connect to 127.0.0.1:58775

[jborbely]

Running the following module on the 32-bit server is a reliable way to simulate the issue you are experiencing

import time
from msl.loadlib import Server32

class HangsForever(Server32):
    def __init__(self, host, port):
        time.sleep(999)

A CTRL+C must be invoked to terminate the 64-bit process and the KeyboardInterrupt exception points to self._proc.wait() on L201 as the culprit.

Your initial suggestion of using self._proc.wait(timeout=timeout) is one way to fix this issue (but it introduces the concerns that I mentioned above). I will investigate possible solutions.

[jborbely]

Thanks for raising this issue. It has been fixed in 1f4971c

[learningmadeeasyfun]

Thanks a lot for the update - can you please let me know how we can pull your fix locally - do we need to reinstall the package. Will all the new installation of the package will have the fix in concern?

Thanks in advance.

[jborbely]

You can re-install the package from the main branch using pip

pip install -U https://github.com/MSLNZ/msl-loadlib/archive/main.tar.gz

Let me know if the issue remains. That means my test didn't cover the issue properly.

[learningmadeeasyfun]

I tried to install the package as per command mentioned but I get an error now:

C:\WINDOWS\system32>pip install -U https://github.com/MSLNZ/msl-loadlib/archive/main.tar.gz
Collecting https://github.com/MSLNZ/msl-loadlib/archive/main.tar.gz
Downloading https://github.com/MSLNZ/msl-loadlib/archive/main.tar.gz | 19.4 MB 6.8 MB/s 
Using legacy 'setup.py install' for msl-loadlib, since package 'wheel' is not installed. 
Installing collected packages: msl-loadlib Attempting uninstall: msl-loadlib 
Found existing installation: msl-loadlib 0.9.0
Uninstalling msl-loadlib-0.9.0:
Successfully uninstalled msl-loadlib-0.9.0
Running setup.py install for msl-loadlib ... done
Successfully installed msl-loadlib-0.9.1.dev0 

Now when I run the program I get below error:

File "C:\Python36\lib\site-packages\msl\loadlib\client64.py", line 147, in __init__
raise OSError('Cannot find ' + server_exe) 
OSError: Cannot find C:\Python36\lib\site-packages\msl\loadlib\server32-windows.exe                                                                                                                           

It seems some component got missed in the package.

[jborbely]

I see that you don't have the wheel package installed

Using legacy 'setup.py install' for msl-loadlib, since package 'wheel' is not installed.

I have pushed a fix when installing the package in a Python environment that does not have the wheel package installed.

Could you try running the following command again

pip install -U https://github.com/MSLNZ/msl-loadlib/archive/main.tar.gz

and let me know what the outcome is.

[learningmadeeasyfun]

I am not sure whether the installation process was success - the same I state is based on previous uninstalling / installing step during installation and recent output as mentioned below:

C:\WINDOWS\system32>pip install -U https://github.com/MSLNZ/msl-loadlib/archive/main.tar.gz
Collecting https://github.com/MSLNZ/msl-loadlib/archive/main.tar.gz
Downloading https://github.com/MSLNZ/msl-loadlib/archive/main.tar.gz
| 19.4 MB 3.2 MB/s  
C:\WINDOWS\system32> 

[jborbely]

What is the output of running these two commands?

pip --version
pip list

[learningmadeeasyfun]

Please find the details as requested:

C:\Users\python>pip --version 
pip 21.1.2 from c:\python36\lib\site-packages\pip (python 3.6)

C:\Users\python>pip list 
Package Version ------------------------------ ---------- 
alabaster 0.7.12
allure-pytest 2.6.2
allure-python-commons 2.6.2
astroid 2.4.2
atomicwrites 1.2.1
attrs 18.2.0
Babel 2.7.0
backports.functools-lru-cache 1.6.1
beautifulsoup4 4.8.1
bitstring 3.1.4
bottle 0.12.19
bs4 0.0.1
certifi 2019.9.11
cffi 1.14.0
chardet 3.0.4
colorama 0.4.0
conan 1.33.1
coverage 5.1
cryptography 2.8
cycler 0.10.0
Cython 0.29.21
deprecation 2.0.7
distro 1.5.0
docutils 0.15.2
elevate 0.1.3
enum34 1.1.10
fasteners 0.16
future 0.16.0
idna 2.8
imagesize 1.1.0
importlib-metadata 1.7.0
isort 4.3.4
Jinja2 2.10
kiwisolver 1.0.1
lazy-object-proxy 1.4.3
lxml 4.4.1
MarkupSafe 1.0
matplotlib 3.1.0
mccabe 0.6.1
mock 2.0.0
more-itertools 4.3.0
MouseInfo 0.1.2
msl-loadlib 0.9.1.dev0
nidaqmx 0.5.7
node-semver 0.6.1
npTDMS 0.18.1
numpy 1.19.5
opcua 0.98.9
packaging 19.2
pandas 1.0.0
patch-ng 1.17.4
pbr 4.3.0
PeakUtils 1.1.1
Pillow 7.0.0
pip 21.1.2
pluggy 0.13.1
pluginbase 1.0.0
prompt-toolkit 1.0.18
protocol 0.1.0
psutil 5.6.0
py 1.7.0
PyAutoGUI 0.9.47
pycparser 2.20
pycryptodomex 3.9.0
PyGetWindow 0.0.8
Pygments 2.4.2
PyJWT 1.7.1
pylint 2.3.1
pymodbus 2.4.0
pyModbusTCP 0.1.6
PyMsgBox 1.0.7
pyodbc 4.0.25
pyparsing 2.2.0
pyperclip 1.7.0
Pypubsub 4.0.3
PyRect 0.1.4
pyreserve 1.0.0a1
PyScreeze 0.1.26
pyserial 3.4
pysoem 1.0.0
pytest 5.4.1
pytest-cov 2.8.1
pytest-mock 1.10.0
python-dateutil 2.7.3
pythonnet 2.4.0
PyTweening 1.0.3
pytz 2018.5
pywin32 224
PyYAML 5.4.1
pyzmq 19.0.1
requests 2.22.0
robotframework 3.1.2
robotframework-debuglibrary 1.2
robotframework-lint 0.9
robotframework-ride 1.7.3.1
robotframework-seleniumlibrary 3.3.1
robotframework-SikuliLibrary 1.0.8
robotframeworklexer 1.1
saleae 0.9.1
scapy 2.4.0
scipy 1.1.0
selenium 3.141.0
setuptools 28.8.0
six 1.15.0
snowballstemmer 2.0.0
soupsieve 1.9.4
Sphinx 1.8.3
sphinxcontrib-robotdoc 0.10.0
sphinxcontrib-websupport 1.1.2
system 0.1.16
tqdm 4.56.2
typed-ast 1.4.1
Typhoon-HIL-API 1.9.0
TyphoonTest 1.10.2
urllib3 1.25.6
utils 0.9.0
wcwidth 0.1.8
win-inet-pton 1.0.1
WMI 1.5.1
wrapt 1.11.2
wxPython 4.0.6
zipp 3.1.0

[jborbely]

According to pip, you have msl-loadlib 0.9.1.dev0 installed. So does your code work?

Here are some commands that you can run to clean up and verify your Python environment.

Your version of setuptools is quite old setuptools 28.8.0, to upgrade it run

pip install -U setuptools

To uninstall msl-loadlib run

pip uninstall --yes msl-loadlib

To install msl-loadlib from the main branch of github

pip install -U https://github.com/MSLNZ/msl-loadlib/archive/main.tar.gz

If you also have git installed then you can use the following command to install msl-loadlib from the main branch of github. Installing the package this way will append the hash value of the commit to the version identifier so that you know what version of the unreleased code you are using.

pip install -U git+https://github.com/MSLNZ/msl-loadlib@main

Verify the installation. You should see the number 2 printed to the terminal if installation was successful

python -c "from msl.examples.loadlib import Cpp64; c=Cpp64(); print(c.add(1, 1))"

To illustrate how to use some of these commands I created a new Python 3.6 environment and then installed the package and verified the installation

C:\Users\user>pip list
Package    Version
---------- -------
pip        21.1.2
setuptools 57.0.0

C:\Users\user>pip install -U git+https://github.com/MSLNZ/msl-loadlib@main
Collecting git+https://github.com/MSLNZ/msl-loadlib@main
  Cloning https://github.com/MSLNZ/msl-loadlib (to revision main) to c:\users\user\appdata\local\temp\pip-req-build-37ul4b3y
  Running command git clone -q https://github.com/MSLNZ/msl-loadlib 'C:\Users\user\AppData\Local\Temp\pip-req-build-37ul4b3y'
Using legacy 'setup.py install' for msl-loadlib, since package 'wheel' is not installed.
Installing collected packages: msl-loadlib
    Running setup.py install for msl-loadlib ... done
Successfully installed msl-loadlib-0.9.1.dev0+fe5ee3e

C:\Users\user>pip list
Package     Version
----------- ------------------
msl-loadlib 0.9.1.dev0+fe5ee3e
pip         21.1.2
setuptools  57.0.0

C:\Users\user>python -c "from msl.examples.loadlib import Cpp64; c=Cpp64(); print(c.add(1, 1))"
2

If after following these suggestion you are still having issues then I'm not sure there is anything else that I can recommend except to reinstall the package from PyPI using pip install -U msl-loadlib and then use your initial suggestion of modifying the code to use self._proc.wait(timeout=timeout) on line 201.

[learningmadeeasyfun]

Thanks - yes the code works as expected - no issues found so far.

Actually I was in concern that the package version that I am working with would be the same when we install in our production or test bench system. It seems that it would not hence requesting changes so that any further installation of package via pip would have the fix. I would then make use of the below mentioned pip command in our other boxes:

pip install -U https://github.com/MSLNZ/msl-loadlib/archive/main.tar.gz

[jborbely]

Thanks for you feedback and for verifying that your code works as expected.