Placeholder for version 4 of config.yml

[jimisola]

Work has started with the syntax for version 3 4 of config.yml.

This issue will be used a placeholder to link with other issues and feature request as well a providing a place for discussion (in addition to slack).

The current draft version of Configuration Syntax v4 can be found here in our wiki.

Requirements:

• YAML 1.2
• highlight use of node anchors
• jinja2 templates
  • built-in filters
  • standard custom filter ENV for environment variables
  • any other custom filter?
  • what’s needed to handle different input data formats for rendering (json, yaml, xml etc)?
• work with document separator “---”
• work with multiple files, e.g. gitlabform file1 file2 file3
• prepare to minor adaptation of format when GitLab merges groups and projects into namespaces
  • can we be backwards-compatible with or will the API change so that there will have to be two versions of gitlabform (pre- [ ] and post-namespaces)?
• work with personal projects
• inheritance:
  • do-not-allow-inherit-overwrite
  • do-not-inherit

Questions:

• Shall we plan for use of python-gitlab?
• Shall we plan for update, add and delete?
  • Currently gitlabform support update with a exception (add/delete) for keys and variables
  • with python-gitlab it should be easier to support add and delete
• How should inheritance be combined with group/project members and “enforce”?
• GitLab mixes update and edit? Shall we use their mix or just settle for one of them? (follow-up: @jimisola )
• members: both project and groups?
  • group_ldap_links: can only be applied to groups or projects as well? only groups
  • What is this for: https://docs.gitlab.com/ee/api/groups.html#create-a-link-to-share-a-group-with-another-group ?
    • Invite members
    • Invite groups
    • sync with LDAP groups
• tags: belongs to project? Yes.
• what terminology to use for variables vs secret_variables vs cicd_variables: variables (@gdubicki this is already in v3 right?)
• keys/names of groups and projects
• Services API → Integrations API: (@gdubicki this is already in v3 right?)

Relevant links:

• Idea for custom YAML resolver (pyyaml): here

[jimisola]

Related issues and PRs are:

• Allow for same "level" to exist multiple times #327
• Allow breaking configuration inheritence #326
• Expand wildcard support #325
• skip_groups doesn't work for subgroup #275
• Use project and group id instead of names #254
• Support for patterns in project and group names #139
• WIP: Proposal of the v4 (?) config syntax #138
• Use "project name" as variable #112
• Group/subgroup processing order matters. #105
• Breaking changes proposals for GitLabForm 2.* #82
• Support multiple config files #13
• Filter projects_and_groups configuration using project topics #398

[amimas]

Since inherit flag is implemented through #326 , maybe skip flag can be deprecated now and be removed in v3 v4? Or, is that still needed?

[jimisola]

I've had some meetings with GitLab through work and I expressed then and in issues that it is important that they keep their APIs up to date with new functionality. Otherwise, that functionality isn't really useful since it's impractical (doesn't scale) to configure groups/projects manually.

I was informed that one issue they have is to manage two APIs (REST and GraphQL). Apparently, they working on some middle layer to be able make it easier. I'll check up on it again.

We also know that GitLab have some major changes coming up ahead e.g. with the introduction of work items (expansion of issues) and Namespaces (merge of projects/groups).

So, the question remains if for v4 we should continue with @gdubicki current solution which we then can assume we'll have to make an effort to keep up to date with all the coming changes or if we should make an effort to switch to python-gitlab in order to not spend time on an own API library?

[lfvjimisola]

Regarding "include" and "exclude" see comment and comment).