Windows cwltool crashes when /tmp does not exist

[bpmweel]

I just installed cwltool under windows using the cwltooldoc.md installation procedure. After installation I got the error trace below.

After creating the /tmp directory the error went away.

Perhaps this can be added to the installation documentation or the tmpdir can be created or set to a different directory based on environment variables.

PS C:\> cwltool
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python\Python3.6\Scripts\cwltool-script.py", line 11, in <module>
    load_entry_point('cwltool==1.0.20170723124118', 'console_scripts', 'cwltool')()
  File "c:\program files (x86)\python\python3.6\lib\site-packages\pkg_resources\__init__.py", line 565, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "c:\program files (x86)\python\python3.6\lib\site-packages\pkg_resources\__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "c:\program files (x86)\python\python3.6\lib\site-packages\pkg_resources\__init__.py", line 2291, in load
    return self.resolve()
  File "c:\program files (x86)\python\python3.6\lib\site-packages\pkg_resources\__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "c:\program files (x86)\python\python3.6\lib\site-packages\cwltool\main.py", line 22, in <module>
    import schema_salad.validate as validate
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "c:\program files (x86)\python\python3.6\lib\site-packages\past\translation\__init__.py", line 341, in load_module
    mod = imp.load_module(fullname, *self.found)
  File "c:\program files (x86)\python\python3.6\lib\imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "c:\program files (x86)\python\python3.6\lib\imp.py", line 172, in load_source
    module = _load(spec)
  File "c:\program files (x86)\python\python3.6\lib\site-packages\schema_salad\validate.py", line 3, in <module>
    import avro.schema
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "c:\program files (x86)\python\python3.6\lib\site-packages\past\translation\__init__.py", line 396, in load_module
    if detect_python2(source, self.pathname):
  File "c:\program files (x86)\python\python3.6\lib\site-packages\past\translation\__init__.py", line 222, in detect_python2
    with open('/tmp/original_code.py', 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/original_code.py'

[kapilkd13]

Thanks @bpmweel for reporting this. I can reproduce the error. I tested this on python2 and python 3. It appears this error occurs only in python3 as the concerned error is raised by past module a supporting module like future which helps in py2/3 compatibility. It allows python 2 functionality on python3 ( oppo of future).
@bpmweel If its possible can you try installing cwltool with python2 and tell us the behaviour you noticed.

Below snippet is from past module which raised this error:

    if source != str(tree)[:-1]:   # remove added newline
        # The above fixers made changes, so we conclude it's Python 2 code
        logger.debug('Detected Python 2 code: {0}'.format(pathname))
        with open('/tmp/original_code.py', 'w') as f:
            f.write('### Original code (detected as py2): %s\n%s' % 
                    (pathname, source))
        with open('/tmp/py2_detection_code.py', 'w') as f:
            f.write('### Code after running py3 detection (from %s)\n%s' % 
                    (pathname, str(tree)[:-1]))
        return True
    else:
        logger.debug('Detected Python 3 code: {0}'.format(pathname))
        with open('/tmp/original_code.py', 'w') as f:
            f.write('### Original code (detected as py3): %s\n%s' % 
                    (pathname, source))
        try:
            os.remove('/tmp/futurize_code.py')
        except OSError:
            pass
        return False

Here tmp is hardcoded.
@manu-chroma Since you have worked on python 3 compatibility, I think you can help me with this error. What do you think about this issue?
For now I would like to drop python 3 support from windows doc. Also I would raise this issue to past module community.

[manu-chroma]

@kapilkd13 Do we have the permission to create tmp folder? If so I can add a snippet to create and remove it, before and after the past module function call.

[kapilkd13]

@manu-chroma That is a option. About permission: On windows tmp folder will be created in the root of you current drive, for ex if your current directory is Drive:\foo\bar then this module assume it to be present in Drive:\tmp.
I tested with

import os
os.makedirs('/tmp')

and it works fine. But in order to avoid race condition

import os
import errno

def make_sure_path_exists(path):
    try:
        os.makedirs(path)
    except OSError as exception:
        if exception.errno != errno.EEXIST:
            raise

this looks like a nice option.
So if we have the permission to create a folder in the root of current drive, this option would work and usually we have this permission.

[manu-chroma]

@kapilkd13 Do we require \tmp folder in running any of our tests?

[kapilkd13]

@manu-chroma Tests that use cwltool main function to execute complete workflow do need this \tmp folder. But since on Windows we need Docker and as its not available on Appveyor for Linux image we have disabled these tests on Appveyor. So Currently no unit tests that are performed on Appveyor need \tmp folder, as we have passed all tests on Appveyor for Python 3. Though Conformance tests would need this folder.

[mr-c]

From today's call: @manu-chroma will create an issue at https://github.com/PythonCharmers/python-future about removing the hardcoding paths from https://github.com/PythonCharmers/python-future/blob/master/src/past/translation/__init__.py#L222 etc

[manu-chroma]

From today's call: manu-chroma will create an issue at PythonCharmers/python-future about removing the hardcoding paths from PythonCharmers/python-future:src/past/translation/init.py@master#L222 etc

@mr-c, @kapilkd13 will be working on the checklist you've created. He found the problem in the first place and is willing to work on it.