Emergent failure in test_apply_pyproject_equivalent_to_setupcfg

[jaraco]

3 days ago, the head commit at 6067fa4 was passing tests, but when re-run here, two tests are failing (two forms of test_apply_pyproject_equivalent_to_setupcfg).

_ test_apply_pyproject_equivalent_to_setupcfg[https://github.com/aio-libs/aiohttp/raw/5e0e6b7080f2408d5f1dd544c0e1cf88378b7b10/setup.cfg] _
[gw1] linux -- Python 3.11.9 /home/runner/work/setuptools/setuptools/.tox/py/bin/python

url = 'https://github.com/aio-libs/aiohttp/raw/5e0e6b7080f2408d5f1dd544c0e1cf88378b7b10/setup.cfg'
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f25c8dc2390>
tmp_path = PosixPath('/tmp/pytest-of-runner/pytest-0/popen-gw1/test_apply_pyproject_equivalen7')

    @pytest.mark.parametrize("url", urls_from_file(HERE / EXAMPLES_FILE))
    @pytest.mark.filterwarnings("ignore")
    @pytest.mark.uses_network
    def test_apply_pyproject_equivalent_to_setupcfg(url, monkeypatch, tmp_path):
        monkeypatch.setattr(expand, "read_attr", Mock(return_value="0.0.1"))
        setupcfg_example = retrieve_file(url)
        pyproject_example = Path(tmp_path, "pyproject.toml")
        setupcfg_text = setupcfg_example.read_text(encoding="utf-8")
        toml_config = Translator().translate(setupcfg_text, "setup.cfg")
        pyproject_example.write_text(toml_config, encoding="utf-8")
    
        dist_toml = pyprojecttoml.apply_configuration(makedist(tmp_path), pyproject_example)
        dist_cfg = setupcfg.apply_configuration(makedist(tmp_path), setupcfg_example)
        _port_tests_require(dist_cfg)
    
        pkg_info_toml = core_metadata(dist_toml)
        pkg_info_cfg = core_metadata(dist_cfg)
>       assert pkg_info_toml == pkg_info_cfg
E       assert 'Metadata-Ver... "speedups"\n' == 'Metadata-Ver... "speedups"\n'
E         
E         Skipping 265 identical leading characters in diff, use -v to show
E         - =workflow%3ACI
E         + =workflow%%3ACI
E         ?           +
E           Project-URL: Coverage: codecov, https://codecov.io/github/aio-libs/aiohttp
E           Project-URL: Docs: Changelog, https://docs.aiohttp.org/en/stable/changes.html...
E         
E         ...Full output truncated (30 lines hidden), use '-vv' to show

setuptools/tests/config/test_apply_pyprojecttoml.py:57: AssertionError
----------------------------- Captured stdout call -----------------------------
Skipping https://github.com/aio-libs/aiohttp/raw/5e0e6b7080f2408d5f1dd544c0e1cf88378b7b10/setup.cfg (already exists: /home/runner/work/setuptools/setuptools/setuptools/tests/config/downloads/aio-libs_aiohttp5e0e6b7080f2408d5f1dd544c0e1cf88378b7b10_setup.cfg)
------------------------------ Captured log call -------------------------------
WARNING  ini2toml.plugins:toml_incompatibilities.py:36 Sections 'flake8' ('setup.cfg') may be problematic.

It might be the case TOML files are not supported by the relevant tools,
or that 'ini2toml' just lacks a plugin for this kind of configuration.

Please review the generated output and remove these sections if necessary.
_ test_apply_pyproject_equivalent_to_setupcfg[https://github.com/sqlalchemy/sqlalchemy/raw/533f5718904b620be8d63f2474229945d6f8ba5d/setup.cfg] _
[gw1] linux -- Python 3.11.9 /home/runner/work/setuptools/setuptools/.tox/py/bin/python

url = 'https://github.com/sqlalchemy/sqlalchemy/raw/533f5718904b620be8d63f2474229945d6f8ba5d/setup.cfg'
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f25ca3b7c10>
tmp_path = PosixPath('/tmp/pytest-of-runner/pytest-0/popen-gw1/test_apply_pyproject_equivalen10')

    @pytest.mark.parametrize("url", urls_from_file(HERE / EXAMPLES_FILE))
    @pytest.mark.filterwarnings("ignore")
    @pytest.mark.uses_network
    def test_apply_pyproject_equivalent_to_setupcfg(url, monkeypatch, tmp_path):
        monkeypatch.setattr(expand, "read_attr", Mock(return_value="0.0.1"))
        setupcfg_example = retrieve_file(url)
        pyproject_example = Path(tmp_path, "pyproject.toml")
        setupcfg_text = setupcfg_example.read_text(encoding="utf-8")
        toml_config = Translator().translate(setupcfg_text, "setup.cfg")
        pyproject_example.write_text(toml_config, encoding="utf-8")
    
>       dist_toml = pyprojecttoml.apply_configuration(makedist(tmp_path), pyproject_example)

setuptools/tests/config/test_apply_pyprojecttoml.py:51: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
setuptools/config/pyprojecttoml.py:68: in apply_configuration
    config = read_configuration(filepath, True, ignore_option_errors, dist)
setuptools/config/pyprojecttoml.py:129: in read_configuration
    validate(subset, filepath)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

config = {'project': {'authors': [{'email': 'mike_mp@zzzcomputing.com', 'name': 'Mike Bayer'}], 'classifiers': ['Development St...cense-files': ['LICENSE'], 'package-dir': {'': 'lib'}, 'packages': {'find': {'namespaces': False, 'where': ['lib']}}}}}
filepath = '/tmp/pytest-of-runner/pytest-0/popen-gw1/test_apply_pyproject_equivalen10/pyproject.toml'

    def validate(config: dict, filepath: StrPath) -> bool:
        from . import _validate_pyproject as validator
    
        trove_classifier = validator.FORMAT_FUNCTIONS.get("trove-classifier")
        if hasattr(trove_classifier, "_disable_download"):
            # Improve reproducibility by default. See issue 31 for validate-pyproject.
            trove_classifier._disable_download()  # type: ignore
    
        try:
            return validator.validate(config)
        except validator.ValidationError as ex:
            summary = f"configuration error: {ex.summary}"
            if ex.name.strip("`") != "project":
                # Probably it is just a field missing/misnamed, not worthy the verbosity...
                _logger.debug(summary)
                _logger.debug(ex.details)
    
            error = f"invalid pyproject.toml config: {ex.name}."
>           raise ValueError(f"{error}\n{summary}") from None
E           ValueError: invalid pyproject.toml config: `project.optional-dependencies.postgresql_asyncpg[0]`.
E           configuration error: `project.optional-dependencies.postgresql_asyncpg[0]` must be pep508

setuptools/config/pyprojecttoml.py:57: ValueError
----------------------------- Captured stdout call -----------------------------
Skipping https://github.com/sqlalchemy/sqlalchemy/raw/533f5718904b620be8d63f2474229945d6f8ba5d/setup.cfg (already exists: /home/runner/work/setuptools/setuptools/setuptools/tests/config/downloads/sqlalchemy_sqlalchemy533f5718904b620be8d63f2474229945d6f8ba5d_setup.cfg)
------------------------------ Captured log call -------------------------------
WARNING  ini2toml.plugins:toml_incompatibilities.py:36 Sections 'flake8' ('setup.cfg') may be problematic.

It might be the case TOML files are not supported by the relevant tools,
or that 'ini2toml' just lacks a plugin for this kind of configuration.

Please review the generated output and remove these sections if necessary.

[jaraco]

Here's what changed between those builds (Python 3.11):

 setuptools main @ diff --unified orig.deps new.deps
--- orig.deps   2024-04-19 05:46:35
+++ new.deps    2024-04-19 05:45:27
@@ -10,10 +10,11 @@
 chardet==5.2.0
 charset-normalizer==3.3.2
 colorama==0.4.6
+ConfigUpdater==3.2
 coverage==7.4.4
 cryptography==42.0.5
 distlib==0.3.8
-docutils==0.20.1
+docutils==0.21.1
 domdf-python-tools==3.8.0.post2
 execnet==2.1.1
 filelock==3.13.4
@@ -26,11 +27,11 @@
 iniconfig==2.0.0
 jaraco.classes==3.4.0
 jaraco.context==5.3.0
-jaraco.develop==8.11.1
+jaraco.develop==8.13.0
 jaraco.env==1.0.0
 jaraco.envs==2.6.0
-jaraco.functools==4.0.0
-jaraco.packaging==9.5.0
+jaraco.functools==4.0.1
+jaraco.packaging==10.0.0
 jaraco.path==3.7.0
 jaraco.text==3.12.0
 jaraco.ui==2.3.0
@@ -69,11 +70,11 @@
 requests==2.31.0
 requests-file==2.0.0
 requests-toolbelt==1.0.0
-ruff==0.3.7
+ruff==0.4.0
 SecretStorage==3.3.3
-setuptools @ file:///home/runner/work/setuptools/setuptools/.tox/.tmp/package/1/setuptools-69.5.1.post20240416-0.editable-py3-none-any.whl#sha256=fadab4f1bd57c44a5fc7cc35cb87e69f42e632072f7bf894c2b33b9c844098c5
+setuptools @ file:///home/runner/work/setuptools/setuptools/.tox/.tmp/package/1/setuptools-69.5.1.post20240418-0.editable-py3-none-any.whl#sha256=4459986076a2dd5055c9f711ede2889dbad1d870c71fb2bd9c28b3d0bcdbecb0
 snowballstemmer==2.2.0
-Sphinx==7.2.6
+Sphinx==7.3.6
 sphinxcontrib-applehelp==1.0.8
 sphinxcontrib-devhelp==1.0.6
 sphinxcontrib-htmlhelp==2.0.5
@@ -85,10 +86,11 @@
 toml==0.10.2
 tomli==2.0.1
 tomli_w==1.0.0
+tomlkit==0.12.4
 tox==4.14.2
 typeguard==4.2.1
 typing_extensions==4.11.0
 urllib3==2.2.1
-virtualenv==20.25.1
-wheel==0.42.0
+virtualenv==20.25.3
+wheel==0.43.0
 zipp==3.18.1

[jaraco]

I'm able to replicate the same failure on my machine locally.

Uninstalling ConfigUpdater allows the tests to pass and reinstalling restores the failure.

[jaraco]

Some Python versions were released recently, so I thought that might be a factor, but both runs are Python 3.11.9.

I don't yet understand where ConfigUpdater comes from or why it's relevant. Searching the codebase, I don't find that string at all.

[jaraco]

I see now. Setuptools tests depend on ini2toml[lite], but late versions of jaraco.develop depend on ini2toml[full] and that's where ConfigUpdater gets pulled in and its presence violates the test assumptions. That dependency was added in jaraco/jaraco.develop@52e113e as part of coherent-oss/roadmap#8.

[jaraco]

I'm unsure the right course of action here. Ideally, the tests shouldn't be sensitive to the ini2toml flavor installed, but they are.

We could remove the setuptools[testing] dependency on jaraco.develop, which is required only to allow tools/finalize.py to be doctested (amounting to an import check at the moment).

For now I'll mark the test as xfail while we investigate/discuss.

[abravalheri]

Thanks @jaraco, for identifying that. I believe I can change that to stop failing (either by explicitly specifying the translator to use ConfigParser or by changing ini2toml directly).

The main difference is that ConfigUpdater is more similar to RawConfigParser than ConfigParser when it comes to parsing escaping sequences in the style %%.

Some time ago, I introduced a test case in setuptools inspired by the aiohttp's setup.cfg. That file contains such %%-style escaping sequence.