Test on separate machine

[Czaki]

Because there is no auditwheel/delocate alternative for windows and there are still some problems collecting libs like matthew-brett/delocate#62

Maybe it is good idea to allow run test on separate instance?
Then cibuildwheel will look to wheelhouse and install wheel instead of build them.

@joerick @YannickJadoul What do you think about this?

[YannickJadoul]

I don't really understand. matthew-brett/delocate#62 is about macOS?
And I don't think integrating this into cibuildwheel is so easy. This is mostly something projects will need to do for themselves, seeing if libraries can be loaded on other Windows machines.

[Czaki]

I don't really understand. matthew-brett/delocate#62 is about macOS?

Yes it shows cases when dealocate does not collect all libraries properly.

And I don't think integrating this into cibuildwheel is so easy. This is mostly something projects will need to do for themselves, seeing if libraries can be loaded on other Windows machines.

I have an idea how to do this, But it need some time, so I prefer to ask first.

[joerick]

Hard to say without knowing - what's your idea? The main cost would be added complexity - if it's simple to implement, it could be worth it.

[Czaki]

idea is to add test switch or separate executable. Then add build test function similar to build
Something like:

def test(options: BuildOptions):
    abs_project_dir = os.path.abspath(options.project_dir)
    temp_dir = tempfile.mkdtemp(prefix='cibuildwheel')
    built_wheel_dir = os.path.join(temp_dir, 'built_wheel')
    repaired_wheel_dir = os.path.join(temp_dir, 'repaired_wheel')

    python_configurations = get_python_configurations(options.build_selector)

    for config in python_configurations:
        dependency_constraint_flags = []
        if options.dependency_constraints:
            dependency_constraint_flags = [
                '-c', options.dependency_constraints.get_for_python_version(config.version)
            ]
        if options.test_command:        
            env = setup_python(config, dependency_constraint_flags, options.environment)
            wheel_path = find_wheel()
           
            # check that we are using the Python from the virtual environment
            call(['which', 'python'], env=env)

            if options.before_test:
                before_test_prepared = prepare_command(options.before_test, project=abs_project_dir)
                call(before_test_prepared, env=env, shell=True)

            # install the wheel
            call(['pip', 'install', wheel_path + options.test_extras], env=env)

            # test the wheel
            if options.test_requires:
                call(['pip', 'install'] + options.test_requires, env=env)

            # run the tests from $HOME, with an absolute path in the command
            # (this ensures that Python runs the tests against the installed wheel
            # and not the repo code)
            test_command_prepared = prepare_command(options.test_command, project=abs_project_dir)
            call(test_command_prepared, cwd=os.environ['HOME'], env=env, shell=True)

[joerick]

interesting... I note that this would change the order of execution - all would build, then all would test. But how would this be exposed to the user?

[Czaki]

I would like not to change execution order, but give possibility to use other build order.
I think about two options separate entry point or add switch to cibuildwheel argument.

[YannickJadoul]

So you're kind of suggestion a conceptually separate citestwheel or so?

I'm wondering if there's a way to disable the "build phase", similar to how we'd now disable the test phase by making CIBW_TEST_COMMAND empty. In principle, you could have a big if that checks if the wheel needs to be built or just install a pre-built from somewhere. (So we'd need a way of providing an alternative package index? Or actually, even download from somewhere that isn't a package index, then?)

I think it's somehow good to immediately test a newly wheel, in the default case, though. You don't want to build everything to then figure out tests are failing?

[Czaki]

The Idea is to allow test on separate machine - not to force them. There are still scenario that not all libs are collected properly (especially on windows). But if this libraries are compiled on build machine all test could pass (because wheel after installation could found lib files).

The libraries could be compiled outside setup.py to save build time for compilation of libs which not depend on python version.

[YannickJadoul]

The Idea is to allow test on separate machine - not to force them.

I know. I was not claiming that, I think.

[YannickJadoul]

Btw, multibuild seems to test some things in a separate container: https://github.com/matthew-brett/multibuild/#test-phase

On a separate note: I know multibuild is credited in the README, but where exactly does cibuildwheel differ so much? To some degree it feels silly doing the same things (like this testing, or updating openssl, or #331) completely independently.

[janaknat]

We are looking to build and test aarch64 wheels using GitHub CI and cibuildwheel (kivy/pyjnius#639). As part of that, there is a requirement to build and test the wheels in separate containers. This is to 'prevent build script/installation from influencing tests (virtualenv is not enough sometimes) - kivy/pyjnius#639 (comment).

Going by the fact that this issue is still open there is no option to allow testing separately from the step that built the wheels?

[Czaki]

Going by the fact that this issue is still open there is no option to allow testing separately from the step that built the wheels?

No, it just means, that no one sits to implement this in a clean way (to avoid code repeatability). I'm currently more focused on other projects. Maybe I will find some time in the future.