Update generates invalid config file

[CarliJoy]

Description of the problem

Trying to update a 3.2 project

setup.cfg

# This file is used to configure your project.
# Read more about the various options under:
# http://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-setup-using-setup-cfg-files

[metadata]
name = pyscaffoldext-company
description =  company specific extensions to allow continuous integration
               and best standards within the company
author = Carli Freudenberg
author-email = Carl.Freudenberg@company.de
license = mit
long-description = file: README.rst
long-description-content-type = text/x-rst; charset=UTF-8
url = https://gitlab.company.de/python-devops/pyscaffoldext-company
project-urls =
    Documentation = https://pypi.company.de/company/stable/pyscaffoldext-company/latest/+d/index.html
# Change if running only on Windows, Mac or Linux (comma-separated)
platforms = any
# Add here all kinds of additional classifiers as defined under
# https://pypi.python.org/pypi?%3Aaction=list_classifiers
classifiers =
    Development Status :: 5 - Production/Stable
    Programming Language :: Python :: 3 :: Only
    Topic :: Software Development :: Libraries :: Python Modules
    Topic :: Software Development :: Code Generators
    Operating System :: OS Independent
    Intended Audience :: Developers

[options]
zip_safe = False
packages = find:
include_package_data = True
package_dir =
    =src
# Fix the pyscaffold version so we have fewer things to test
install_requires =
    pyscaffold==3.2.3
    pyscaffoldext-pyproject
    black>=20.8b1
# DON'T CHANGE THE FOLLOWING LINE! IT WILL BE UPDATED BY PYSCAFFOLD!
setup_requires = pyscaffold>=3.2a0,<3.3a0
# Add here dependencies of your project (semicolon/line-separated), e.g.
# install_requires = numpy; scipy
# The usage of test_requires is discouraged, see `Dependency Management` docs
# tests_require = pytest; pytest-cov
# Require a specific Python version, e.g. Python 2.7 or >= 3.4
python_requires = >=3.6

[options.packages.find]
where = src
exclude =
    tests

[options.extras_require]
# Add here additional requirements for extra features, to install with:
# `pip install pyscaffoldext-company[PDF]` like:
# PDF = ReportLab; RXP
# Add here test requirements (semicolon/line-separated)
testing =
    flake8
    pytest
    pytest-cov
    pytest-virtualenv
    pytest-xdist

build_docu =
    sphinx
    recommonmark

[options.entry_points]
pyscaffold.cli =
    company = pyscaffoldext.company.extension:company
# Add here console scripts like:
# console_scripts =
#     script_name = pyscaffoldext.company.module:function
# For example:
# console_scripts =
#     fibonacci = pyscaffoldext.company.skeleton:run
# And any other entry points, for example:
# pyscaffold.cli =
#     awesome = pyscaffoldext.awesome.extension:AwesomeExtension

[test]
# py.test options when running `python setup.py test`
# addopts = --verbose
extras = True

[tool:pytest]
# Options for py.test:
# Specify command line options as you would do when invoking py.test directly.
# e.g. --cov-report html (or xml) for html/xml output or --junitxml junit.xml
# in order to write a coverage file that can be read by Jenkins.
addopts =
    --cov pyscaffoldext.company --cov-report term-missing
    --verbose
norecursedirs =
    dist
    build
    .tox
testpaths = tests
# Use pytest markers to select/deselect specific tests
# markers =
#     slow: mark tests as slow (deselect with '-m "not slow"')

[aliases]
dists = bdist_wheel

[bdist_wheel]
# Use this option if your package is pure-python
universal = 1

[build_sphinx]
source_dir = docs
build_dir = build/sphinx

[devpi:upload]
# Options for the devpi: PyPI server and packaging tool
# VCS export must be deactivated since we are using setuptools-scm
no-vcs = 1
formats = bdist_wheel

[flake8]
# Some sane defaults for the code style checker flake8
max-line-length = 88
extend-ignore = E203, W503
# ^  Black-compatible
#    E203 and W503 have edge cases handled by black
exclude =
    .tox
    build
    dist
    .eggs
    docs/conf.py

[pyscaffold]
# PyScaffold's parameters when the project was created.
# This will be used when updating. Do not change!
version = 3.2.3.post0.dev58+g6145ac0
package = company
extensions =
    no_skeleton
    namespace
    pre_commit
    tox
    travis
    custom_extension

Traceback.

$ putup --version
PyScaffold 4.1

$ putup --update --force .
Traceback (most recent call last):
  File "c:\company\python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\company\python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\user\.local\bin\putup.exe\__main__.py", line 7, in <module>
  File "C:\Users\user\.local\pipx\venvs\pyscaffold\lib\site-packages\pyscaffold\shell.py", line 113, in func_wrapper
    func(*args, **kwargs)
  File "C:\Users\user\.local\pipx\venvs\pyscaffold\lib\site-packages\pyscaffold\exceptions.py", line 33, in func_wrapper
    func(*args, **kwargs)
  File "C:\Users\user\.local\pipx\venvs\pyscaffold\lib\site-packages\pyscaffold\cli.py", line 263, in run
    main(args or sys.argv[1:])
  File "C:\Users\user\.local\pipx\venvs\pyscaffold\lib\site-packages\pyscaffold\cli.py", line 256, in main
    opts["command"](opts)
  File "C:\Users\user\.local\pipx\venvs\pyscaffold\lib\site-packages\pyscaffold\cli.py", line 225, in run_scaffold
    api.create_project(opts)
  File "C:\Users\user\.local\pipx\venvs\pyscaffold\lib\site-packages\pyscaffold\api.py", line 156, in create_project
    return reduce(actions.invoke, pipeline, ({}, opts))
  File "C:\Users\user\.local\pipx\venvs\pyscaffold\lib\site-packages\pyscaffold\actions.py", line 98, in invoke
    return action(*struct_and_opts)
  File "C:\Users\user\.local\pipx\venvs\pyscaffold\lib\site-packages\pyscaffold\update.py", line 66, in version_migration
    return reduce(invoke, plan_actions, (struct, opts))
  File "C:\Users\user\.local\pipx\venvs\pyscaffold\lib\site-packages\pyscaffold\actions.py", line 98, in invoke
    return action(*struct_and_opts)
  File "C:\Users\user\.local\pipx\venvs\pyscaffold\lib\site-packages\pyscaffold\update.py", line 77, in _wrapped
    setupcfg.update_file()
  File "C:\Users\user\.local\pipx\venvs\pyscaffold\lib\site-packages\configupdater\configupdater.py", line 172, in update_file
    self.validate_format(**self._parser_opts)
  File "C:\Users\user\.local\pipx\venvs\pyscaffold\lib\site-packages\configupdater\document.py", line 97, in validate_format
    parser.read_string(str(self))
  File "c:\company\python38\lib\configparser.py", line 723, in read_string
    self.read_file(sfile, source)
  File "c:\company\python38\lib\configparser.py", line 718, in read_file
    self._read(f, source)
  File "c:\company\python38\lib\configparser.py", line 1113, in _read
    raise e
configparser.ParsingError: Source contains parsing errors: '<string>'
        [line 147]: 'namespace\n'

Please provide any additional information below.

Versions and main components

• PyScaffold Version: 4.1
• Python Version: 3.8.7
• Operating system: Windows 10 20H2
• How did you install PyScaffold: pipx install pyscaffold

[abravalheri]

Thank you very much for the report @CarliJoy, I will investigate this as soon as I have some time.

Could you please also do a pipx runpip pyscaffold freeze? (It might be a problem with configupdater version).

[CarliJoy]

appdirs==1.4.4
ConfigUpdater==3.0
packaging==21.0
pyparsing==2.4.7
PyScaffold==4.1
setuptools-scm==6.3.2
tomli==1.2.1
tomlkit==0.7.2

[abravalheri]

Thanks @CarliJoy.

I investigated a little bit and the problem seems to be caused by a conjunction of factors:

1. The venv created for pipx does not have pyscaffoldext-custom-extension (I recommend using pipx runpip pyscaffold install pyscaffoldext-custom-extension pyscaffoldext-travis or install pyscaffold[all])
2. In PyScaffold < 4.1 there was a bug that prevented persisting the namespace = pyscaffoldext in the [pyscaffold] section of setup.cfg. Consequently, PyScaffold is loading the configuration as if namespace = None (pyscaffoldext-custom-extension should set it correctly, but unfortunately it is not currently installed).
3. ConfigUpdater is persisting namespace = None as the string "namespace" instead of "namespace =", and this seems to be incompatible with ConfigParser (used for the validation).

So to fix the issue in the long run, I think we need to fix ConfigUpdater first. We also need to put some warnings/errors/prints to tell the user when to install missing extensions.

But on the bright side I believe that if you install the extensions (pipx runpip pyscaffold install pyscaffoldext-custom-extension pyscaffoldext-travis or install pyscaffold[all]), and try again, you should not see this error.

There is a second bug: currently PyScaffold assumes the description field in setup.cfg is a single line. As a workaround I also recommend collapsing your 2 lines into a single one before trying to run the update.

Please let me know if these workarounds temporarily solve your problem. I will try to address both issues in patch releases for PyScaffold and ConfigUpdater in the near future.

[abravalheri]

@CarliJoy, there is a new RC: https://pypi.org/project/PyScaffold/4.1.1rc1/ that supposedly fix this issue.

Please let me know your feedback.

[abravalheri]

I will release the final 4.1.1 with the bugfix for this issue.
Please feel free to reopen the issue if the problem persists.

[CarliJoy]

@abravalheri thank you very much :-)

And Sorry for my missing feedback. I tested it a bit (and it seems to be working) but had to time to a proper test.
As I am about to leave the company I am currently working for (and in which this was used) - I had quite a bit of higher prioritized jobs to do.
And I my free time I was missing the data :(

[abravalheri]

No problems @CarliJoy, those things happen all the time.
I tend to leave issues open for a while, so people have the chance to provide feedback for the proposed solution, but then after a bit I just assume things are working well.

I wish you all the best in the new journey!

[abravalheri]

Hi @CarliJoy, just a heads up, Setuptools recently dropped support for "multi-line short description", details:

• Newlines in the description field produce a malformed PKG-INFO pypa/setuptools#1390 (comment)
• Fail on a multiline distribution package summary pypa/setuptools#2870
• https://packaging.python.org/specifications/core-metadata/#summary

So, while PyScaffold functionality is unaffected, you might have problems with your builds.

[CarliJoy]

Thanks for the reference. Very thoughtful :-)