Meet 'FileNotFoundError' on git-bash/Windows 10

[ccli8]

Environment

Windows 10 64-bit
Python 3.7
Git for Windows 2.25.0 64-bit
manifest-tool 1.5.2

Description

I am running Pelion E2E tests. For non-update test, it is passed:

pytest TESTS/pelion-e2e-python-test-library/tests/dev-client-tests.py \
-k test_01_get_device_id

For update test, I meet FileNotFoundError.

pytest TESTS/pelion-e2e-python-test-library/tests/dev-client-tests.py \
-k test_06_update_device \
--update_bin=/C/CCLi8/IoT/mbed/mbed-examples/mbed-os-example-pelion/BUILD/NUMAKER_PFM_NUC472/ARMC6/mbed-os-example-pelion_update.bin \
--manifest_tool=/C/CCLi8/IoT/mbed/mbed-examples/mbed-os-example-pelion

Per preliminary check, the update image upload step has succeeded. The failure point is in manifest-tool create. I check my manifest-tool installation directory and find there's no manifest-tool.exe (only manifest-tool python script) there. I am not sure if this is the cause on Windows environment.

Log

=================================== ERRORS ====================================
___________________ ERROR at setup of test_06_update_device ___________________

cloud = <pelion_test_lib.cloud.cloud.PelionCloud object at 0x000001701AAC6C50>
client = <pelion_test_lib.tools.client_runner.Client object at 0x000001701B3BDC50>
request = <SubRequest 'update_device' for <Function test_06_update_device>>

    @pytest.fixture(scope='function')
    def update_device(cloud, client, request):
        """
        Fixture for updating device.
        This uploads firmware image, creates manifest, uploads manifest, and creates and starts the update campaign.
        :param cloud: Cloud fixture
        :param client: Client fixture
        :param request: Requests fixture
        :return: Campaign ID
        """
        binary_path = request.config.getoption('update_bin', None)
        manifest_tool_path = request.config.getoption('manifest_tool', os.getcwd())
        no_cleanup = request.config.getoption('no_cleanup', False)
        delta_manifest = request.config.getoption('delta_manifest', None)
        log.info('Update image: "{}"'.format(binary_path))
        log.info('Path for manifest-tool init: "{}"'.format(manifest_tool_path))
        if not binary_path or not manifest_tool_path:
            skip_msg = 'Provide missing binary image in startup arguments to run update case\n' \
                       '--update_bin={}'.format(binary_path if binary_path else 'MISSING!')
            log.warning(skip_msg)
            pytest.skip(skip_msg)
    
        sleep(5)
        resp = cloud.device_directory.get_device(client.endpoint_id(), expected_status_code=200).json()
        assert resp['state'] == 'registered'
    
        fw_image = cloud.update.upload_firmware_image(binary_path, expected_status_code=201).json()
        fw_image_id = fw_image['id']
        log.info('Firmware image uploaded! Image ID: {}'.format(fw_image_id))
    
        manifest_file = manifest_tool.create_manifest(path=manifest_tool_path,
                                                      firmware_url=fw_image['datafile'],
                                                      update_image_path=binary_path,
>                                                     delta_manifest=delta_manifest)

..\..\mbed-libs\pelion-e2e-python-test-library\pelion_test_lib\fixtures\cloud_fixtures.py:138: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
TESTS\pelion-e2e-python-test-library\pelion_test_lib\tools\manifest_tool.py:92: in create_manifest
    p = subprocess.Popen(cmd, cwd=path, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
c:\python37\lib\subprocess.py:775: in __init__
    restore_signals, start_new_session)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <subprocess.Popen object at 0x000001701B3EB940>
args = 'manifest-tool create -u http://firmware-catalog-media-ca57.s3.dualstack.us-east-1.amazonaws.com/mbed-os-example-pelio...:/CCLi8/IoT/mbed/mbed-examples/mbed-os-example-pelion/BUILD/NUMAKER_PFM_NUC472/ARMC6/mbed-os-example-pelion_update.bin'
executable = None, preexec_fn = None, close_fds = False, pass_fds = ()
cwd = 'C:\\CCLi8\\IoT\\mbed\\mbed-examples\\mbed-os-example-pelion', env = None
startupinfo = <subprocess.STARTUPINFO object at 0x000001701B3EB0B8>
creationflags = 0, shell = False, p2cread = Handle(900), p2cwrite = -1
c2pread = 12, c2pwrite = Handle(704), errread = 13, errwrite = Handle(1000)
unused_restore_signals = True, unused_start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       unused_restore_signals, unused_start_new_session):
        """Execute program (MS Windows version)"""
    
        assert not pass_fds, "pass_fds not supported on Windows."
    
        if not isinstance(args, str):
            args = list2cmdline(args)
    
        # Process startup details
        if startupinfo is None:
            startupinfo = STARTUPINFO()
        else:
            # bpo-34044: Copy STARTUPINFO since it is modified above,
            # so the caller can reuse it multiple times.
            startupinfo = startupinfo._copy()
    
        use_std_handles = -1 not in (p2cread, c2pwrite, errwrite)
        if use_std_handles:
            startupinfo.dwFlags |= _winapi.STARTF_USESTDHANDLES
            startupinfo.hStdInput = p2cread
            startupinfo.hStdOutput = c2pwrite
            startupinfo.hStdError = errwrite
    
        attribute_list = startupinfo.lpAttributeList
        have_handle_list = bool(attribute_list and
                                "handle_list" in attribute_list and
                                attribute_list["handle_list"])
    
        # If we were given an handle_list or need to create one
        if have_handle_list or (use_std_handles and close_fds):
            if attribute_list is None:
                attribute_list = startupinfo.lpAttributeList = {}
            handle_list = attribute_list["handle_list"] = \
                list(attribute_list.get("handle_list", []))
    
            if use_std_handles:
                handle_list += [int(p2cread), int(c2pwrite), int(errwrite)]
    
            handle_list[:] = self._filter_handle_list(handle_list)
    
            if handle_list:
                if not close_fds:
                    warnings.warn("startupinfo.lpAttributeList['handle_list'] "
                                  "overriding close_fds", RuntimeWarning)
    
                # When using the handle_list we always request to inherit
                # handles but the only handles that will be inherited are
                # the ones in the handle_list
                close_fds = False
    
        if shell:
            startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
            startupinfo.wShowWindow = _winapi.SW_HIDE
            comspec = os.environ.get("COMSPEC", "cmd.exe")
            args = '{} /c "{}"'.format (comspec, args)
    
        # Start the process
        try:
            hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                                     # no special security
                                     None, None,
                                     int(not close_fds),
                                     creationflags,
                                     env,
                                     os.fspath(cwd) if cwd is not None else None,
>                                    startupinfo)
E                                    FileNotFoundError: [WinError 2] The system cannot find the file specified

c:\python37\lib\subprocess.py:1178: FileNotFoundError

[teetak01]

@janisimonen @calmjm can you check this?

[janisimonen]

Hi @ccli8!
Were you able to run manifest-tool init before starting the pytest?

[ccli8]

@janisimonen , on my git-bash/Windows 10, I can run manifest-tool init separately and successfully. Actually, before the pytest, I've run mbed dev-mgmt init, which includes manifest-tool init.

[janisimonen]

Okey, interesting. I'll try to find a Win machine and try to reproduce this situation here.

[soleilplanet]

pelion_e2e_test.log
same issue observed when running this test for DISCO_L496AG + STMod BG96
Wondering if this section of log is relevant to the problem:

def _execute_child(self, args, executable, preexec_fn, close_fds,
                   pass_fds, cwd, env,
                   startupinfo, creationflags, shell,
                   p2cread, p2cwrite,
                   c2pread, c2pwrite,
                   errread, errwrite,
                   unused_restore_signals, unused_start_new_session):
    """Execute program (MS Windows version)"""

    assert not pass_fds, "pass_fds not supported on Windows."

[janisimonen]

Hi @ccli8 and @soleilplanet!
Looks like the (pip) installed manifest-tool is missing from the Windows' systems PATH - if it's not found by the pytest. So, if you can check where the manifest-tool is installed and then add that folder to env PATH - e.g like this: https://www.architectryan.com/2018/03/17/add-to-the-path-on-windows-10/

[ccli8]

@janisimonen I am sure manifest-tool has installed in my Windows. Checking my python installation path C:\Python37, I find the file manifest-tool is placed under C:\Python37\Scripts, but no manifest-tool.exe. Actually, I find no manifest-tool.exe in my Windows.

[janisimonen]

@ccli8 Looks like manifest-tool has issue on pip installation - I just noted the same behaviour. So, you need to clone the manifest-tool repo and install the local copy on Windows environment to have executable where it should be.
PelionIoT/manifest-tool#11

[ccli8]

@janisimonen The workaround works. I find manifest-tool.exe under C:\Python37\Scripts folder now. Thanks for your information.

[janisimonen]

@ccli8 Good to hear - thanks!