New version 8.x.x ignores GL_TOKEN

[Sieboldianus]

The problem

I had a working pyproject.toml for automatic versioning in CI:

[tool.semantic_release]
version_variable = ".version:__version__"
upload_to_pypi = "false"
build_command = "false"
branch = "main"
version_source = "tag"
commit_version_number = "true"
upload_to_repository = "false"
hvcs = "gitlab"
hvcs_domain = "gitlab.hrz.tu-chemnitz.de"
hvcs_api_domain = "gitlab.hrz.tu-chemnitz.de/api/v4"
env = "GL_TOKEN"

(I added env = "GL_TOKEN" per the new docs)

This does not work anymore. I get the following result in CI:

if [ -z ${GL_TOKEN+x} ]; then echo "GL_TOKEN is unset"; else echo "GL_TOKEN is set"; fi
GL_TOKEN is set
$ semantic-release version
[12:06:14] WARNING  [semantic_release.cli.config] WARNING          config.py:331
                    config.from_raw_config: the token for the                   
                    remote VCS is configured as stored in the                   
                    GH_TOKEN environment variable, but it is empty  

Expected behavior

Semantic-release should pickup Gitlab, not Github as HVCS.

Environment

I tested this locally, and get the same reuslts:

semantic-release --noop -v version

Output

semantic-release --noop -v version
🛡 You are running in no-operation mode, because the '--noop' flag was supplied
[14:08:39] INFO     [semantic_release.cli.util] INFO util.load_raw_config_file: Loading configuration from pyproject.toml                                                                                  util.py:84
           INFO     [semantic_release.cli.config] INFO config.select_branch_options: Using group 'main' options, as '(main|master)' matches 'main'                                                      config.py:228
           INFO     [semantic_release.version.algorithm] INFO algorithm.tags_and_versions: found 16 previous tags                                                                                     algorithm.py:56
           INFO     [semantic_release.version.algorithm] INFO algorithm.next_version: Found 16 full releases (excluding prereleases)                                                                 algorithm.py:246
           INFO     [semantic_release.version.algorithm] INFO algorithm.next_version: The last full release was 0.9.4, tagged as 'v0.9.4'                                                            algorithm.py:268
           INFO     [semantic_release.version.algorithm] INFO algorithm.bfs: found latest version in branch history: '0.9.4' (5d8e7cd)                                                                algorithm.py:92
           INFO     [semantic_release.version.algorithm] INFO algorithm._bfs_for_latest_version_in_history: the latest version in this branch's history is 0.9.4                                     algorithm.py:111
           INFO     [semantic_release.version.algorithm] INFO algorithm.next_version: The last full version in this branch's history was 0.9.4                                                       algorithm.py:292
           INFO     [semantic_release.version.algorithm] INFO algorithm.next_version: The type of the next release release is: minor                                                                 algorithm.py:369
0.10.0
           INFO     [semantic_release.version.algorithm] INFO algorithm.tags_and_versions: found 16 previous tags                                                                                     algorithm.py:56
The next version is: 0.10.0! 🚀
🛡 semantic-release 'noop' mode is enabled! would have updated versions in the following paths:
🛡 semantic-release 'noop' mode is enabled! would have run the build_command false
🛡 semantic-release 'noop' mode is enabled!
    would have run:
        git add
        git commit -m "0.10.0

            Automatically generated by python-semantic-release"

           INFO     [semantic_release.version.algorithm] INFO algorithm.tags_and_versions: found 16 previous tags                                                                                     algorithm.py:56
[14:08:40] INFO     [semantic_release.cli.commands.version] INFO version.version: Path 'templates' not found, using default changelog template                                                         version.py:386
🛡 semantic-release 'noop' mode is enabled! would have written your changelog to CHANGELOG.md
🛡 semantic-release 'noop' mode is enabled!
    would have run:
        git add CHANGELOG.md

           INFO     [semantic_release.cli.commands.version] INFO version.version: No local changes to add to any commit, skipping                                                                      version.py:447
🛡 semantic-release 'noop' mode is enabled!
    would have run:
        git tag -a v0.10.0 -m "v0.10.0"

🛡 semantic-release 'noop' mode is enabled!
    would have run:
        git push https://****@github.com/ad/test.git main
        git push --tags https://****@github.com/ad/test.git main

🛡 semantic-release 'noop' mode is enabled! would have created a release for the tag 'v0.10.0'
🛡 semantic-release 'noop' mode is enabled! would have uploaded the following assets: []
           INFO     [semantic_release.cli.github_actions_output] INFO github_actions_output.write_if_possible: not writing GitHub Actions output, as no file specified                    github_actions_output.py:70

python --version
> Python 3.8.12
pip --version
> pip 21.3.1
semantic-release --version
> semantic-release, version 8.0.3

pip freeze

bleach==4.1.0
build==0.10.0
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.10
click==8.0.3
click-log==0.3.2
colorama==0.4.4
cryptography==36.0.1
docutils==0.18.1
dotty-dict==1.3.0
gitdb==4.0.9
GitPython==3.1.26
idna==3.3
importlib-metadata==4.10.1
importlib-resources==5.7.0
invoke==1.6.0
jeepney==0.7.1
Jinja2==3.1.2
keyring==23.5.0
markdown-it-py==3.0.0
MarkupSafe==2.1.3
mdurl==0.1.2
packaging==21.3
pkginfo==1.8.2
pycparser==2.21
pydantic==1.10.12
Pygments==2.15.1
pyparsing==3.0.6
pyproject_hooks==1.0.0
python-gitlab==2.10.1
python-semantic-release==8.0.3
readme-renderer==32.0
requests==2.27.1
requests-toolbelt==0.9.1
rfc3986==2.0.0
rich==13.4.2
SecretStorage==3.3.1
semver==2.13.0
setuptools-scm==6.4.1
shellingham==1.5.0.post1
six==1.16.0
smmap==5.0.0
tomli==2.0.0
tomlkit==0.10.2
tqdm==4.62.3
twine==3.7.1
typing_extensions==4.7.1
urllib3==1.26.8
webencodings==0.5.1
zipp==3.7.0

[Sieboldianus]

I solved my problem. What helped me was generating a default config file with

semantic-release generate-config -f toml --pyproject >> pyproject.toml

I merged these with my custom variables. Here is the configuration that works for me with the current Gitlab version:

[tool.semantic_release]
assets = []
commit_message = "{version}\n\nAutomatically generated by python-semantic-release"
commit_parser = "angular"
logging_use_named_masks = false
major_on_zero = true
tag_format = "v{version}"
version_source = "tag"
commit_version_number = "true"
branch = "main"
version_variable = ".version:__version__"

[tool.semantic_release.branches.main]
match = "(main)"
prerelease = false
prerelease_token = "rc"

[tool.semantic_release.changelog]
template_dir = "templates"
changelog_file = "CHANGELOG.md"
exclude_commit_patterns = []

[tool.semantic_release.changelog.environment]
block_start_string = "{%"
block_end_string = "%}"
variable_start_string = "{{"
variable_end_string = "}}"
comment_start_string = "{#"
comment_end_string = "#}"
trim_blocks = false
lstrip_blocks = false
newline_sequence = "\n"
keep_trailing_newline = false
extensions = []
autoescape = true

[tool.semantic_release.commit_author]
env = "GIT_COMMIT_AUTHOR"
default = "semantic-release <semantic-release>"

[tool.semantic_release.commit_parser_options]
allowed_tags = ["build", "chore", "ci", "docs", "feat", "fix", "perf", "style", "refactor", "test"]
minor_tags = ["feat"]
patch_tags = ["fix", "perf"]

[tool.semantic_release.remote]
name = "origin"
type = "gitlab"
ignore_token_for_push = false
hvcs = "gitlab"
hvcs_domain = "gitlab.hrz.tu-chemnitz.de"
hvcs_api_domain = "gitlab.hrz.tu-chemnitz.de/api/v4"

[tool.semantic_release.remote.token]
env = "GL_TOKEN"

[tool.semantic_release.publish]
dist_glob_patterns = []
upload_to_vcs_release = false
upload_to_pypi = false
build_command = false
upload_to_repository = false

..and this is the corresponding Gitlab CI

(skip the parts that are not relevant to you)

stages:
  - build
  - deploy

build:
  image: "registry.gitlab.vgiscience.org/tud_ifk/miniconda-cidefault:0.2.0"
  stage: build
  variables:
      GIT_FETCH_EXTRA_FLAGS: --tags
      GITLAB_CI : "true"
  before_script:
       # initialize conda shell
      - conda init bash
      - source ~/.bashrc
      # activate default ci_env from registry image
      # contains anybadge
      - conda activate ci_env
      - git fetch --tags --force
  script:
      # checkout to release branch
      - git checkout -B "$CI_COMMIT_REF_NAME" "$CI_COMMIT_SHA"
      - git config --global user.name "semantic-release (via GitlabCI)"
      - git config --global user.email "semantic-release@gitlab"
      # create release version
      - url_host=`git remote get-url origin | sed -e "s/https:\/\/gitlab-ci-token:.*@//g"`
      - git remote set-url origin "https://gitlab-ci-token:${GL_TOKEN}@${url_host}"
      - if [ -z ${GL_TOKEN+x} ]; then echo "GL_TOKEN is unset"; else echo "GL_TOKEN is set"; fi
      - semantic-release version
      - semantic-release publish
      - git push origin main --follow-tags
      # create badges with version from latest tag
      - git fetch --tags
      - version_str=$(git describe --tags $(git rev-list --tags --max-count=1))
      - version_var=$(echo $version_str | cut -c2-)
      - echo "Version for badges $version_var"
      - anybadge -l version --value="${version_var}" --file=version.svg --color=green
      # collect artifacts
      - mkdir ._site
      - cp *.svg ._site
      - cp -r resources/* ._site
      - mv ._site _site
  artifacts:
    name: _site
    paths:
      - _site
  only:
    - main

deploy:
  stage: deploy
  image: "registry.gitlab.vgiscience.org/tud_ifk/rsync-ssh-alpine:alpine3.17-r0-v4"
  before_script:
  - eval $(ssh-agent -s)
  - echo "${SSH_PRIVKEY}" | ssh-add -
  - mkdir -p ~/.ssh
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
  script:
  - rsync -avu -zz --no-perms --omit-dir-times --chown=www-data:www-data --chmod=D775,F664 _site/ "${PRODUCTION_DESTINATION}"ad/wip/ephemeral_events/
  only:
    - main

What was difficult to me was a) reading the python-semantic-release documentation, it seems inclomplete, particularly regarding the Gitlab configuration and b) identifying deprecated configuration variables. For instance, hvcs, hvcs_domain, and hvcs_api_domain are not mentioned anymore in latest/configuration, but can be found via search (documentation) somewhere nested in the code. They still seem to work and required to get a custom Gitlab CI working. Also, in the migrating_from_v7, it is not mentioned that the env variable GL_TOKEN, which formerly apparently identified Gitlab-CI, does not work anymore and it is required to explicitly add:

[tool.semantic_release.remote]
name = "origin"
type = "gitlab"

It is also unclear to me why so many sections in pyproject.toml are needed. Why not allow everything under [tool.semantic_release]. All this configuration (for a single tool) really blows the pyproject.toml. Finally, it is unclear what configuration entries can be safely removed because they are the default behaviour.

Importantly, this is not meant to be a ranting! I really, really appreciate all the work that is going on in python-semantic-release. I just hope I can contribute to identify issues with the documentation, for better onboarding of people just starting with the tool.

[bernardcooke53]

Hey @Sieboldianus 👋

Glad you were able to get up and running - but thank you for a) leaving such a detailed description of how you were able to resolve the issue and b) for highlighting your pain points 🥇

Definitely room to improve on the documentation - if you find anything else that seems missing, unclear or wrong, please don't hesitate to point it out!

I've put together #665 and #666 for the documentation issues you've highlighted here 🙏

[Sieboldianus]

Hi @bernardcooke53, of course, glad I could contribute at least a bit here.

I will see if I can create a PR with a minimal Gitlab CI Guide. I've been using python-semantic-release with Gitlab CI since 3-4 years and it worked flawlessly.

There was one other piece that left me clueless while migrating from 7 to 8, perhaps you know better where to put this: There's now a new category "Unknown" showing up in my releases. I often have commits that are "dirty", i.e. 20x Fix ci, or a quick Fix typo - I know this is my fault, but I am often the only contributor and these are sometimes even private repos (I version everything). It was great in 7.x.x not having these dirty commits showing in my release changelog. I looked at exclude_commit_patterns = [] but the documentation does not say how to exclude the Unknown category. Maybe this is not possible.. For the exclude_commit_patterns, an example could really help formatting this option (this applies to the rest of the documentation, too - I seem to understand parameter use best where example values are given).

Otherwise, 8.x.x works like 7.x.x before now, really happy 👍 I will jump in those issues and see if I can add more details later. Thank you for picking this up.

[Edit]

For anyone looking.. I've cleaned up my gitlab-ci.yml posted above after I fixed and tested my workflow with 8.x.x:

Click

stages:
  - build
  - deploy

build:
  image: "registry.gitlab.vgiscience.org/tud_ifk/miniconda-cidefault:0.2.0"
  stage: build
  variables:
      GIT_FETCH_EXTRA_FLAGS: --tags
  before_script:
       # initialize conda shell
      - conda init bash
      - source ~/.bashrc
      # activate default ci_env from registry image
      # contains anybadge
      - conda activate ci_env
  script:
      # checkout to release branch
      - git checkout -B "$CI_COMMIT_REF_NAME" "$CI_COMMIT_SHA"
      - git config --global user.name "semantic-release (via GitlabCI)"
      - git config --global user.email "semantic-release@gitlab"
      # create release version
      - url_host=`git remote get-url origin | sed -e "s/https:\/\/gitlab-ci-token:.*@//g"`
      - git remote set-url origin "https://gitlab-ci-token:${GL_TOKEN}@${url_host}"
      - semantic-release version
      - semantic-release publish
      - git push origin main --follow-tags
      # create badges with version from latest tag
      - version_str=$(git describe --tags $(git rev-list --tags --max-count=1))
      - version_var=$(echo $version_str | cut -c2-)
      - echo "Version for badges $version_var"
      - anybadge -l version --value="${version_var}" --file=version.svg --color=green
      # collect artifacts
      - mkdir ._site
      - cp *.svg ._site
      - cp -r resources/* ._site
      - mv ._site _site
  artifacts:
    name: _site
    paths:
      - _site
  only:
    - main

deploy:
  stage: deploy
  image: "registry.gitlab.vgiscience.org/tud_ifk/rsync-ssh-alpine:alpine3.17-r0-v4"
  before_script:
  - eval $(ssh-agent -s)
  - echo "${SSH_PRIVKEY}" | ssh-add -
  - mkdir -p ~/.ssh
  - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
  script:
  - rsync -avu -zz --no-perms --omit-dir-times --chown=www-data:www-data --chmod=D775,F664 _site/ "${PRODUCTION_DESTINATION}"ad/wip/ephemeral_events/
  only:
    - main

I think the following lines are also not necessary any more and can be removed (but I haven't tested):

• GIT_FETCH_EXTRA_FLAGS: --tags - git checkout should do this automatically and tags are normally available
• git push origin main --follow-tags - semantic-release should publish to git automatically; since I am not publishing, another option could be to remove semantic-release publish