problems with setting variables which already exist (idempotence) with gitlabform 2.10.1 #354

weakcamel · 2022-04-08T14:47:48Z

Describe the bug

This issue appeared when I bumped gitlabform from 2.8.1 to 2.10.1 (also visible in 2.10.0)

❯ gitlabform --strict -v -c secrets.yml very-silly-test-group-for-ldap-and-fun
🏗  GitLabForm version: 2.10.0 = outdated 😔, please update! (the latest stable is 2.10.1)
Reading config from file: secrets.yml
Connected to GitLab version: 13.12.15-ee (e4445d58bff)
>>> Getting requested groups/projects...
:: # of groups to process: 1
groups: ['very-silly-test-group-for-ldap-and-fun']
:: # of projects to process: 0
:: (# of omitted projects - empty effective config: 1)
projects: []
omitted projects - empty effective config: ['very-silly-test-group-for-ldap-and-fun/testy-project']
@ (1/1) Processing group: very-silly-test-group-for-ldap-and-fun
Processing section 'group_secret_variables'
Adding build_vault_addr of group_secret_variables in very-silly-test-group-for-ldap-and-fun
Warning: Error occurred while processing group very-silly-test-group-for-ldap-and-fun, exception:

Request url='https://mygitlab.example.com/api/v4/groups/very-silly-test-group-for-ldap-and-fun/variables', method=POST, data='{"key": "VAULT_ADDR", "value": "https://vault.example.com"}' failed - expected code(s) [201], got code 400 & body: '{"message":{"key":["(VAULT_ADDR) has already been taken"]}}'

Traceback (most recent call last):
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/core.py", line 365, in run
    self.group_processors.process_entity(
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/processors/__init__.py", line 30, in process_entity
    processor.process(
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/processors/util/decorators.py", line 42, in method_wrapper
    return method(self, project_and_group, SafeDict(configuration), *args)
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/processors/abstract_processor.py", line 48, in process
    self._process_configuration(project_or_project_and_group, configuration)
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/processors/multiple_entities_processor.py", line 113, in _process_configuration
    self.add_method(project_or_group, entity_config)
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/gitlab/group_secret_variables.py", line 11, in post_group_secret_variable
    self._make_requests_to_api(
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/gitlab/core.py", line 130, in _make_requests_to_api
    response = self._make_request_to_api(
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/gitlab/core.py", line 221, in _make_request_to_api
    raise UnexpectedResponseException(
gitlabform.gitlab.core.UnexpectedResponseException: Request url='https://mygitlab.example.com/api/v4/groups/very-silly-test-group-for-ldap-and-fun/variables', method=POST, data='{"key": "VAULT_ADDR", "value": "https://vault.example.com"}' failed - expected code(s) [201], got code 400 & body: '{"message":{"key":["(VAULT_ADDR) has already been taken"]}}'

If I delete the group variable, this call adds it back but fails on re-run.

GitLabForm version

Output of gitlabform -V

❯ gitlabform -V
🏗  GitLabForm version: 2.10.1 = the latest stable 😊

GitLab version

13.12.15-ee

The text was updated successfully, but these errors were encountered:

gdubicki · 2022-04-15T19:11:06Z

Hi @weakcamel!

I tested it on gitlabform 2.10.0 + GitLab 14.8.5-ee, gitlabform 2.10.1 + GitLab 14.2.4-ee and gitlabform 2.10.1 + GitLab 13.12.15-ee and I wasn't able to reproduce it.

gdubicki · 2022-04-21T19:28:58Z

Can you please retry with the latest pre-release version, enable debug mode and share the output, @weakcamel? Note: please check that output for secrets before sharing!

weakcamel · 2022-04-22T08:32:57Z

Hello,

Sorry for late reply, I've been away for a couple of days. Interesting that it doesn't always reproduce.. hmm...

~~By pre-release version do you mean the latest from main branch or are there pre-built pre-release versions somewhere?~~

weakcamel · 2022-04-22T08:44:58Z

Installed the latest version from main

pip install git+https://github.com/gdubicki/gitlabform.git

and the output (redacted) is

weakcamel ~/git/config (master U:1*) $ gitlabform -c secrets.yml --strict --debug   very-silly-test-group-for-ldap-and-fun
🏗  GitLabForm version: 2.11.0b1 = pre-release 🤩 (the latest stable is 2.10.1)
Reading config from file: secrets.yml
Config parsed successfully as YAML.
Starting new HTTPS connection (1): gitlab.example.io:443
https://gitlab.example.io:443 "GET /api/v4/version HTTP/1.1" 200 50
<--- json = {"revision": "e4445d58bff", "version": "13.12.15-ee"}
Connected to GitLab version: 13.12.15-ee (e4445d58bff)
>>> Getting requested groups/projects...
https://gitlab.example.io:443 "GET /api/v4/groups/very-silly-test-group-for-ldap-and-fun HTTP/1.1" 200 None
<--- json = {"auto_devops_enabled": null, "avatar_url": null, "created_at": "2018-07-16T12:17:19.219Z", "default_branch_protection": 3, "description": "", "emails_disabled": null, "extra_shared_runners_minutes_limit": null, "full_name": "very-silly-test-group-for-ldap-and-fun", "full_path": "very-silly-test-group-for-ldap-and-fun", "id": 96, "ldap_access": 20, "ldap_cn": "ldapgroup", "ldap_group_links": [{"cn": "ldapgroup", "filter": null, "group_access": 20, "provider": "ldapmain"}, {"cn": "admin", "filter": null, "group_access": 40, "provider": "ldapmain"}], "lfs_enabled": true, "marked_for_deletion_on": null, "mentions_disabled": null, "name": "very-silly-test-group-for-ldap-and-fun", "parent_id": null, "path": "very-silly-test-group-for-ldap-and-fun", "prevent_forking_outside_group": false, "project_creation_level": "developer", "projects": [{"_links": {"events": "https://gitlab.example.io/api/v4/projects/382/events", "issues": "https://gitlab.example.io/api/v4/projects/382/issues", "labels": "https://gitlab.example.io/api/v4/projects/382/labels", "members": "https://gitlab.example.io/api/v4/projects/382/members", "merge_requests": "https://gitlab.example.io/api/v4/projects/382/merge_requests", "repo_branches": "https://gitlab.example.io/api/v4/projects/382/repository/branches", "self": "https://gitlab.example.io/api/v4/projects/382"}, "allow_merge_on_skipped_pipeline": null, "analytics_access_level": "enabled", "approvals_before_merge": 1, "archived": false, "auto_cancel_pending_pipelines": "enabled", "auto_devops_deploy_strategy": "continuous", "auto_devops_enabled": false, "autoclose_referenced_issues": true, "avatar_url": null, "build_coverage_regex": null, "build_timeout": 3600, "builds_access_level": "enabled", "can_create_merge_request_in": true, "ci_config_path": null, "ci_default_git_depth": null, "ci_forward_deployment_enabled": null, "compliance_frameworks": [], "container_registry_enabled": true, "created_at": "2018-09-17T16:49:14.994Z", "creator_id": 50, "default_branch": "master", "description": "This is my project", "emails_disabled": null, "empty_repo": false, "external_authorization_classification_label": null, "forking_access_level": "enabled", "forks_count": 0, "http_url_to_repo": "https://gitlab.example.io/very-silly-test-group-for-ldap-and-fun/testy-project.git", "id": 382, "import_status": "none", "issues_access_level": "enabled", "issues_enabled": true, "issues_template": null, "jobs_enabled": true, "last_activity_at": "2020-03-12T10:49:35.389Z", "lfs_enabled": true, "marked_for_deletion_at": null, "marked_for_deletion_on": null, "merge_method": "merge", "merge_requests_access_level": "enabled", "merge_requests_enabled": true, "merge_requests_template": null, "mirror": false, "name": "testy-project", "name_with_namespace": "very-silly-test-group-for-ldap-and-fun / testy-project", "namespace": {"avatar_url": null, "full_path": "very-silly-test-group-for-ldap-and-fun", "id": 96, "kind": "group", "name": "very-silly-test-group-for-ldap-and-fun", "parent_id": null, "path": "very-silly-test-group-for-ldap-and-fun", "web_url": "https://gitlab.example.io/groups/very-silly-test-group-for-ldap-and-fun"}, "only_allow_merge_if_all_discussions_are_resolved": true, "only_allow_merge_if_pipeline_succeeds": true, "open_issues_count": 0, "operations_access_level": "enabled", "packages_enabled": null, "pages_access_level": "public", "path": "testy-project", "path_with_namespace": "very-silly-test-group-for-ldap-and-fun/testy-project", "printing_merge_request_link_enabled": true, "public_jobs": true, "readme_url": "https://gitlab.example.io/very-silly-test-group-for-ldap-and-fun/testy-project/-/blob/master/README.md", "remove_source_branch_after_merge": null, "repository_access_level": "enabled", "repository_storage": "default", "request_access_enabled": true, "requirements_enabled": false, "resolve_outdated_diff_discussions": false, "restrict_user_defined_variables": false, "security_and_compliance_enabled": true, "service_desk_address": null, "service_desk_enabled": false, "shared_runners_enabled": true, "shared_with_groups": [{"expires_at": null, "group_access_level": 10, "group_full_path": "teams/ansible", "group_id": 182, "group_name": "ansible"}, {"expires_at": null, "group_access_level": 10, "group_full_path": "teams/security", "group_id": 180, "group_name": "security"}], "snippets_access_level": "disabled", "snippets_enabled": false, "ssh_url_to_repo": "git@gitlab.example.io:very-silly-test-group-for-ldap-and-fun/testy-project.git", "star_count": 0, "suggestion_commit_message": null, "tag_list": [], "visibility": "internal", "web_url": "https://gitlab.example.io/very-silly-test-group-for-ldap-and-fun/testy-project", "wiki_access_level": "disabled", "wiki_enabled": false}], "request_access_enabled": false, "require_two_factor_authentication": false, "runners_token": "u7-YETANOTHERTOKEN", "share_with_group_lock": false, "shared_projects": [], "shared_runners_minutes_limit": null, "shared_with_groups": [], "subgroup_creation_level": "owner", "two_factor_grace_period": 48, "visibility": "internal", "web_url": "https://gitlab.example.io/groups/very-silly-test-group-for-ldap-and-fun"}
https://gitlab.example.io:443 "GET /api/v4/groups/very-silly-test-group-for-ldap-and-fun/projects?include_subgroups=true&per_page=100 HTTP/1.1" 200 None
<--- json = [{"_links": {"events": "https://gitlab.example.io/api/v4/projects/382/events", "issues": "https://gitlab.example.io/api/v4/projects/382/issues", "labels": "https://gitlab.example.io/api/v4/projects/382/labels", "members": "https://gitlab.example.io/api/v4/projects/382/members", "merge_requests": "https://gitlab.example.io/api/v4/projects/382/merge_requests", "repo_branches": "https://gitlab.example.io/api/v4/projects/382/repository/branches", "self": "https://gitlab.example.io/api/v4/projects/382"}, "allow_merge_on_skipped_pipeline": null, "analytics_access_level": "enabled", "approvals_before_merge": 1, "archived": false, "auto_cancel_pending_pipelines": "enabled", "auto_devops_deploy_strategy": "continuous", "auto_devops_enabled": false, "autoclose_referenced_issues": true, "avatar_url": null, "build_coverage_regex": null, "build_timeout": 3600, "builds_access_level": "enabled", "can_create_merge_request_in": true, "ci_config_path": null, "ci_default_git_depth": null, "ci_forward_deployment_enabled": null, "compliance_frameworks": [], "container_registry_enabled": true, "created_at": "2018-09-17T16:49:14.994Z", "creator_id": 50, "default_branch": "master", "description": "This is my project", "emails_disabled": null, "empty_repo": false, "external_authorization_classification_label": null, "forking_access_level": "enabled", "forks_count": 0, "http_url_to_repo": "https://gitlab.example.io/very-silly-test-group-for-ldap-and-fun/testy-project.git", "id": 382, "import_status": "none", "issues_access_level": "enabled", "issues_enabled": true, "issues_template": null, "jobs_enabled": true, "last_activity_at": "2020-03-12T10:49:35.389Z", "lfs_enabled": true, "marked_for_deletion_at": null, "marked_for_deletion_on": null, "merge_method": "merge", "merge_requests_access_level": "enabled", "merge_requests_enabled": true, "merge_requests_template": null, "mirror": false, "name": "testy-project", "name_with_namespace": "very-silly-test-group-for-ldap-and-fun / testy-project", "namespace": {"avatar_url": null, "full_path": "very-silly-test-group-for-ldap-and-fun", "id": 96, "kind": "group", "name": "very-silly-test-group-for-ldap-and-fun", "parent_id": null, "path": "very-silly-test-group-for-ldap-and-fun", "web_url": "https://gitlab.example.io/groups/very-silly-test-group-for-ldap-and-fun"}, "only_allow_merge_if_all_discussions_are_resolved": true, "only_allow_merge_if_pipeline_succeeds": true, "open_issues_count": 0, "operations_access_level": "enabled", "packages_enabled": null, "pages_access_level": "public", "path": "testy-project", "path_with_namespace": "very-silly-test-group-for-ldap-and-fun/testy-project", "printing_merge_request_link_enabled": true, "public_jobs": true, "readme_url": "https://gitlab.example.io/very-silly-test-group-for-ldap-and-fun/testy-project/-/blob/master/README.md", "remove_source_branch_after_merge": null, "repository_access_level": "enabled", "repository_storage": "default", "request_access_enabled": true, "requirements_enabled": false, "resolve_outdated_diff_discussions": false, "restrict_user_defined_variables": false, "security_and_compliance_enabled": true, "service_desk_address": null, "service_desk_enabled": false, "shared_runners_enabled": true, "shared_with_groups": [{"expires_at": null, "group_access_level": 10, "group_full_path": "teams/ansible", "group_id": 182, "group_name": "ansible"}, {"expires_at": null, "group_access_level": 10, "group_full_path": "teams/security", "group_id": 180, "group_name": "security"}], "snippets_access_level": "disabled", "snippets_enabled": false, "ssh_url_to_repo": "git@gitlab.example.io:very-silly-test-group-for-ldap-and-fun/testy-project.git", "star_count": 0, "suggestion_commit_message": null, "tag_list": [], "visibility": "internal", "web_url": "https://gitlab.example.io/very-silly-test-group-for-ldap-and-fun/testy-project", "wiki_access_level": "disabled", "wiki_enabled": false}]
Common config: {"group_secret_variables": {"build_rt_apikey": {"key": "rt_apikey", "value": "REDACTEDTOKEN"}, "build_rt_password": {"key": "rt_password", "value": "abc123"}, "build_rt_user": {"key": "rt_user", "value": "builduser"}, "build_vault_addr": {"key": "VAULT_ADDR", "value": "https://vault.example.io"}, "build_vault_token": {"key": "VAULT_TOKEN", "value": "OTHER-REDACTED-TOKEN"}}}
Group config: {}
Common config: {"group_secret_variables": {"build_rt_apikey": {"key": "rt_apikey", "value": "REDACTEDTOKEN"}, "build_rt_password": {"key": "rt_password", "value": "abc123"}, "build_rt_user": {"key": "rt_user", "value": "builduser"}, "build_vault_addr": {"key": "VAULT_ADDR", "value": "https://vault.example.io"}, "build_vault_token": {"key": "VAULT_TOKEN", "value": "OTHER-REDACTED-TOKEN"}}}
Effective group/subgroup config: {}
Project config: {}
Effective config common+group/subgroup: {"group_secret_variables": {"build_rt_apikey": {"key": "rt_apikey", "value": "REDACTEDTOKEN"}, "build_rt_password": {"key": "rt_password", "value": "abc123"}, "build_rt_user": {"key": "rt_user", "value": "builduser"}, "build_vault_addr": {"key": "VAULT_ADDR", "value": "https://vault.example.io"}, "build_vault_token": {"key": "VAULT_TOKEN", "value": "OTHER-REDACTED-TOKEN"}}}
Effective config common+group/subgroup+project: {"group_secret_variables": {"build_rt_apikey": {"key": "rt_apikey", "value": "REDACTEDTOKEN"}, "build_rt_password": {"key": "rt_password", "value": "abc123"}, "build_rt_user": {"key": "rt_user", "value": "builduser"}, "build_vault_addr": {"key": "VAULT_ADDR", "value": "https://vault.example.io"}, "build_vault_token": {"key": "VAULT_TOKEN", "value": "OTHER-REDACTED-TOKEN"}}}
:: # of groups to process: 1
groups: ['very-silly-test-group-for-ldap-and-fun']
:: # of projects to process: 0
:: (# of omitted projects - empty effective config: 1)
projects: []
omitted projects - empty effective config: ['very-silly-test-group-for-ldap-and-fun/testy-project']
@ (1/1) Processing group: very-silly-test-group-for-ldap-and-fun
Processing section 'group_secret_variables'
https://gitlab.example.io:443 "GET /api/v4/groups/very-silly-test-group-for-ldap-and-fun/variables HTTP/1.1" 200 None
<--- json = [{"environment_scope": "*", "key": "artifactory_API", "masked": false, "protected": false, "value": "https://artifactory.example.io/example/api", "variable_type": "env_var"}, {"environment_scope": "*", "key": "artifactory_NAMESPACE", "masked": false, "protected": false, "value": "example", "variable_type": "env_var"}, {"environment_scope": "*", "key": "artifactory_PROTOCOL", "masked": false, "protected": false, "value": "https", "variable_type": "env_var"}, {"environment_scope": "*", "key": "artifactory_TOKEN", "masked": false, "protected": false, "value": "REDACTEDTOKEN", "variable_type": "env_var"}, {"environment_scope": "*", "key": "artifactory_URL", "masked": false, "protected": false, "value": "artifactory.example.io", "variable_type": "env_var"}, {"environment_scope": "*", "key": "artifactory_USER", "masked": false, "protected": false, "value": "builduser", "variable_type": "env_var"}, {"environment_scope": "*", "key": "DOCKER_DATA_URL", "masked": false, "protected": false, "value": "docker-data.artifactory.example.io", "variable_type": "env_var"}, {"environment_scope": "*", "key": "DOCKER_DEVELOP_URL", "masked": false, "protected": false, "value": "docker-develop.artifactory.example.io", "variable_type": "env_var"}, {"environment_scope": "*", "key": "DOCKER_FEATURE_URL", "masked": false, "protected": false, "value": "docker-feature.artifactory.example.io", "variable_type": "env_var"}, {"environment_scope": "*", "key": "DOCKER_MASTER_URL", "masked": false, "protected": false, "value": "docker-master.artifactory.example.io", "variable_type": "env_var"}, {"environment_scope": "*", "key": "DOCKER_PUBLIC_URL", "masked": false, "protected": false, "value": "docker-public.artifactory.example.io", "variable_type": "env_var"}, {"environment_scope": "*", "key": "DOCKER_SCRATCH_URL", "masked": false, "protected": false, "value": "docker-scratch.artifactory.example.io", "variable_type": "env_var"}, {"environment_scope": "*", "key": "PYPI_DEVELOP_API_URL", "masked": false, "protected": false, "value": "https://artifactory.example.io/example/api/pypi/pypi-develop", "variable_type": "env_var"}, {"environment_scope": "*", "key": "PYPI_DEVELOP_INDEX_URL", "masked": false, "protected": false, "value": "https://artifactory.example.io/example/api/pypi/pypi-develop/simple", "variable_type": "env_var"}, {"environment_scope": "*", "key": "PYPI_DEVELOP_PUBLISH_URL", "masked": false, "protected": false, "value": "https://artifactory.example.io/example/pypi-develop", "variable_type": "env_var"}, {"environment_scope": "*", "key": "PYPI_FEATURE_API_URL", "masked": false, "protected": false, "value": "https://artifactory.example.io/example/api/pypi/pypi-feature", "variable_type": "env_var"}, {"environment_scope": "*", "key": "PYPI_FEATURE_INDEX_URL", "masked": false, "protected": false, "value": "https://artifactory.example.io/example/api/pypi/pypi-feature/simple", "variable_type": "env_var"}, {"environment_scope": "*", "key": "PYPI_FEATURE_PUBLISH_URL", "masked": false, "protected": false, "value": "https://artifactory.example.io/example/pypi-feature", "variable_type": "env_var"}, {"environment_scope": "*", "key": "PYPI_MASTER_API_URL", "masked": false, "protected": false, "value": "https://artifactory.example.io/example/api/pypi/pypi-master", "variable_type": "env_var"}, {"environment_scope": "*", "key": "PYPI_MASTER_INDEX_URL", "masked": false, "protected": false, "value": "https://artifactory.example.io/example/api/pypi/pypi-master/simple", "variable_type": "env_var"}]
group_secret_variables BEFORE: ^^^
Adding build_vault_addr of group_secret_variables in very-silly-test-group-for-ldap-and-fun
https://gitlab.example.io:443 "POST /api/v4/groups/very-silly-test-group-for-ldap-and-fun/variables HTTP/1.1" 400 59
===> data = {"key": "VAULT_ADDR", "value": "https://vault.example.io"}
Warning: Error occurred while processing group very-silly-test-group-for-ldap-and-fun, exception:

Request url='https://gitlab.example.io/api/v4/groups/very-silly-test-group-for-ldap-and-fun/variables', method=POST, data='{"key": "VAULT_ADDR", "value": "https://vault.example.io"}' failed - expected code(s) [201], got code 400 & body: '{"message":{"key":["(VAULT_ADDR) has already been taken"]}}'

Traceback (most recent call last):
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/core.py", line 367, in run
    self.group_processors.process_entity(
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/processors/__init__.py", line 30, in process_entity
    processor.process(
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/processors/util/decorators.py", line 42, in method_wrapper
    return method(self, project_and_group, SafeDict(configuration), *args)
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/processors/abstract_processor.py", line 48, in process
    self._process_configuration(project_or_project_and_group, configuration)
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/processors/multiple_entities_processor.py", line 113, in _process_configuration
    self.add_method(project_or_group, entity_config)
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/gitlab/group_secret_variables.py", line 11, in post_group_secret_variable
    self._make_requests_to_api(
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/gitlab/core.py", line 130, in _make_requests_to_api
    response = self._make_request_to_api(
  File "/Users/weakcamel/git/config/.venv/lib/python3.9/site-packages/gitlabform/gitlab/core.py", line 221, in _make_request_to_api
    raise UnexpectedResponseException(
gitlabform.gitlab.core.UnexpectedResponseException: Request url='https://gitlab.example.io/api/v4/groups/very-silly-test-group-for-ldap-and-fun/variables', method=POST, data='{"key": "VAULT_ADDR", "value": "https://vault.example.io"}' failed - expected code(s) [201], got code 400 & body: '{"message":{"key":["(VAULT_ADDR) has already been taken"]}}'

@ (1/1) FINISHED Processing group: very-silly-test-group-for-ldap-and-fun
:: # of groups processed successfully: 0
:: # of projects processed successfully: 0
:: # of groups failed: 1
:: Failed group 1: very-silly-test-group-for-ldap-and-fun

gdubicki · 2022-04-22T10:32:00Z

Thanks.

It's super strange this output shows that there is no VAULT_ADDR variable in the variables already set in the very-silly-test-group-for-ldap-and-fun group...

Perhaps you have VAULT_ADDR defined as an instance-level variable, @weakcamel?

Update: I did a test with instance-level variable and if it is set it DOES NOT cause error like above for me. But as I don't have any other ideas I would appreciate if you can test at least this...

weakcamel · 2022-04-22T13:57:53Z

I thought it might be a conflict between group and project vars but it's not that either - the project doesn't have any variables at all. Another thought though: there are quite a few variables (30). Perhaps the response gets paginated by Gitlab and there are other pages to be processed?

Update: indeed, when I ran a curl with

curl --header "PRIVATE-TOKEN: $GITLAB_APIKEY"  https://gitlab.example.io/api/v4/groups/very-silly-test-group-for-ldap-and-fun/variables?per_page=100

I could see all the variables whereas without per_page, last variables were stripped off.

as it happened a few times already that we didn't mark an API as using pagination initially and it caused problems for some users. According to GitLab's API docs (https://docs.gitlab.com/ee/api/#pagination) the pagination "is available on all endpoints", so let's just pass the for all GETs. Fixes #354.

gdubicki · 2022-04-23T08:11:54Z

Congrats and thanks for debugging this successfully @weakcamel. 👏😊

I plan to merge the above MR and release a patch release after the weekend.

(Technically this is a bugfix but some people's gitlabform runs may behave differently after we apply it and I try to follow the principle of least astonishment - especially during the weekend. 😇)

as it happened a few times already that we didn't mark an API as using pagination initially and it caused problems for some users. According to GitLab's API docs (https://docs.gitlab.com/ee/api/#pagination) the pagination "is available on all endpoints", so let's just pass the for all GETs. Fixes #354.

gdubicki · 2022-04-25T07:22:32Z

I have released the fix in v2.11.0b2 but it would take me some time to release v2.10.2 with this thing too though.

Can you temporarily switch to using the latest pre-release or exactly v2.11.0b2, @weakcamel, and verify if it works for you?

I expect a final release of v2.11.0 to happen within a few days.

weakcamel · 2022-04-25T13:58:39Z

That's fantastic @gdubicki , many thanks!

I tried the v2.11.0b2 release and it worked like a charm.

gdubicki · 2022-05-09T06:55:02Z

The fix is now in a final release v2.11.0.

gdubicki added the 🐛bug label Apr 11, 2022

gdubicki self-assigned this Apr 15, 2022

gdubicki mentioned this issue Apr 23, 2022

Always expect pagination for GETs #358

Merged

gdubicki closed this as completed in #358 Apr 25, 2022

gdubicki reopened this Apr 25, 2022

gdubicki closed this as completed May 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

problems with setting variables which already exist (idempotence) with gitlabform 2.10.1 #354

problems with setting variables which already exist (idempotence) with gitlabform 2.10.1 #354

weakcamel commented Apr 8, 2022 •

edited

gdubicki commented Apr 15, 2022

gdubicki commented Apr 21, 2022

weakcamel commented Apr 22, 2022 •

edited

weakcamel commented Apr 22, 2022 •

edited

gdubicki commented Apr 22, 2022 •

edited

weakcamel commented Apr 22, 2022 •

edited

gdubicki commented Apr 23, 2022

gdubicki commented Apr 25, 2022

weakcamel commented Apr 25, 2022

gdubicki commented May 9, 2022

problems with setting variables which already exist (idempotence) with gitlabform 2.10.1 #354

problems with setting variables which already exist (idempotence) with gitlabform 2.10.1 #354

Comments

weakcamel commented Apr 8, 2022 • edited

gdubicki commented Apr 15, 2022

gdubicki commented Apr 21, 2022

weakcamel commented Apr 22, 2022 • edited

weakcamel commented Apr 22, 2022 • edited

gdubicki commented Apr 22, 2022 • edited

weakcamel commented Apr 22, 2022 • edited

gdubicki commented Apr 23, 2022

gdubicki commented Apr 25, 2022

weakcamel commented Apr 25, 2022

gdubicki commented May 9, 2022

weakcamel commented Apr 8, 2022 •

edited

weakcamel commented Apr 22, 2022 •

edited

weakcamel commented Apr 22, 2022 •

edited

gdubicki commented Apr 22, 2022 •

edited

weakcamel commented Apr 22, 2022 •

edited