Make source/env discovery extensible

[greg-hellings]

I have a plugin that's been running fine under tox 3. It auto-discovers code on disk and creates environments to manage testing of that code. These tox environments are auto-generated by the plugin.

In tox4, I can move options and manipulation of the work into a subcommand. I have most of the extra plugin functionality mapped to the various plugin hooks.

However, in current tox3 land, I load the auto-discovered tox_envs during the tox_configure hook. There doesn't seem to be an equivalent hook currently in tox4. If I could hook into somewhere in the Source or other environment loading, then I can translate my plugin over to tox4 in preparation for release.

[gaborbernat]

Plugins are totally unsupported at the moment and will not work at all. From plugins point of view version 4 will be totally breaking. We'll define our new plugin interface later this month.

[greg-hellings]

Plugins are totally unsupported at the moment and will not work at all. From plugins point of view version 4 will be totally breaking. We'll define our new plugin interface later this month.

I get that. I'm just putting in my wishlist for items that I would like to see included in the plugin interface! :)

[gaborbernat]

@greg-hellings can you please post a link to your plugin, or present more in-depth how it works? Thanks!

[greg-hellings]

@gaborbernat tox-ansible. As for how it works, it latches into the existing hooks and auto-generates tox environments to run Ansible tests through either the ansible-test command or the molecule command. These are detected by identifying certain files on the filesystem under toxinidir. It also provides the opportunity to filter the generated list of environments based on the attributes of the test environments it discovers. I use this at work to eliminate the need to manually curate a list of tox environments for our CI pipelines.

Things like the filtering for discoverability can, in tox4, easily be added as a sub-command tox ansible. However, generating the environments into the State object currently does not have defined hooks. If a hook to define finding environments could be added to the interface, then I could just register a function to create those environments (e.g. by returning them from the call, or manipulating a collection or the State object, etc).

[ajkavanagh]

Really interested in this too. We (as in the openstack-charmers team) 'borrowed' the ideas from @greg-hellings 's plugin to do a tox 3 plugin for buck, and then I did a tox 4 plugin for buck here: openstack-charmers/buck#9. The tox 4 side was much easier, but it was fairly painful coming up with a system that would cope with both!

I'd be interested in helping with tox on the plugin side, especially as it will affect our plugin :) For the tox 4 plugin, we just hooked into the ini parser, and then got tox to do all the heavy lifting around resolving the env settings.

[gaborbernat]

I think this is where you're getting it wrong. You're not supposed to have a plugin that supports both 3 and 4. tox 3 is deprecated and let it die. This ticket is resolved with the release of v4.

[ajkavanagh]

I think this is where you're getting it wrong. You're not supposed to have a plugin that supports both 3 and 4. tox 3 is deprecated and let it die. This ticket is resolved with the release of v4.

Yes, I understand that tox 3 is deprecated, but I have to support stable versions of code that go back years and have pinned their dependencies. e.g. on python 3.5, 3.6, 3.8. As a tox developer, tox 3 may be dead to you, but I'll be dealing with it for years to come. Hence this plugin started as tox 3, now supports tox 4, and frankly, the tox 4 support is much easier that tox 3 was, which is a bonus.

Eventually, I'll rip out the tox 3 support and make it tox 4 only. That day is not quite here, sadly. However, I'd like to follow the tox 4 plugin evolution, and make sure that our plugin stays on track with what tox 4 is doing. I don't mind helping, or would be happy to test new plugin API features, etc. Thanks.

[gaborbernat]

I don't think you need to support eol packages. People using old versions can use older versions of your plugin. tox 4 supports all non EOL Python's.

[greg-hellings]

@ajkavanagh For supporting both versions of tox, I am able to do it by leveraging import exceptions. The bulk of my code is the same, as it stems from the structure of the Ansible and Molecule code itself. The hooks are the only place where the code really, significantly differs. Prior to the release of tox v4, Sorin put the check in to the tox-ansible code. You can see the catch here. That allows code above it to interface with tox v3 and below it to interface with tox v4. Obviously I'll tidy that up in the final version that includes full v4 support rather than just the exception throwing.

[ajkavanagh]

@greg-hellings I did roughly the same thing, but went in a slightly different direction. I used import exceptions to detect which version of tox, like you, but I made all of the config tox 4 style, and then converted it into tox 3 for backwards compatibility. See https://github.com/openstack-charmers/buck/blob/main/buck/tox_hooks/plugin_hook.py#L25 -- feel free to appropriate any code you see! The tox 4 side simply feeds text into the Ini parser in tox 4. From the tox 3 side, it uses the code that your plugin uses to turn the text into appropriate tox 3 structures. We couldn't have done buck without your work!

[gaborbernat]

You probably know given the comment but you're doing the wrong thing on tox 4, accessing a private variable from tox 4, https://github.com/openstack-charmers/buck/blob/main/buck/tox_hooks/tox_hooks_4.py#L31-L36.

You really should be doing something like

#2882 (comment)