mirror-tool

A tool for maintaining Git subtree mirrors.

• mirror-tool
  • Installation
  • Usage
    • mirror-tool validate-config
    • mirror-tool update-local
    • mirror-tool update
    • mirror-tool promote
    • mirror-tool gitlab-ci-yml
  • Configuration
    • Jinja context
  • License

Installation

Install the mirror-tool package from PyPI.

pip install mirror-tool

Alternatively, run mirror-tool directly as a container image.

Note that you must expose the git repo you want to manage as a directory to the container as the current working directory, as well as setting any needed environment variables if GitLab features are enabled. The details of how to do this are beyond the scope of this README, but an example command may look like:

podman run \
  -v $PWD:/workdir:Z -w /workdir \
  quay.io/rmcgover/mirror-tool:latest \
  update-local

Usage

mirror-tool has the following subcommands. Please install the tool and run with --help for complete documentation on the available commands and options.

mirror-tool validate-config

Verify that .mirror-tool.yaml in the current directory, or a specified configuration file, is valid.

Exits with a 0 exit code if and only if a valid config file was found.

mirror-tool update-local

For each mirror defined in the config file, create a subtree merge commit updating that mirror.

By default, this will not create any commits if there are no changes to be made. It can be forced to create a commit by using the --allow-empty argument.

mirror-tool update

Perform the same updates as update-local, but also push the commit(s) to any configured remote targets.

Currently, GitLab is the only supported target. See the configuration reference below for more information about GitLab integration.

When using this command to push to GitLab, it is recommended to run it from within a GitLab CI/CD pipeline. The command will use predefined environment variables in the CI environment to determine how to connect to GitLab. If used in other contexts, it will be necessary to explicitly set many environment variables.

mirror-tool promote

For any merge requests previously created by update, create additional merge request(s) to promote the same changes to other branch(es) as defined in config.

This command can be used to implement a multi-tiered deployment/update workflow, for example:

• Whenever mirrored repos change, create an MR updating them (via mirror-tool update), targeting testing branch.
• Perform some pre- or post-merge testing on that MR by some means (outside the scope of mirror-tool).
• After the MR is submitted to testing branch, create a new MR promoting the same changes to stable branch (via mirror-tool promote).

The command only operates on changes previously created via mirror-tool update.

Like update, GitLab is currently the only supported target for this command.

mirror-tool gitlab-ci-yml

Generates a .gitlab-ci.yml snippet with a recommended configuration for integrating mirror-tool into a GitLab pipeline.

It is recommended to put the generated config into its own YAML file and use the include keyword to include that file.

The output of this command is influenced by the .mirror-tool.yaml config. When changing configuration elements relating to GitLab, it is a good idea to re-run this command.

Configuration

mirror-tool requires a configuration file. By convention, this should be placed at .mirror-tool.yaml at the top level of your superproject repository.

The following example demonstrates the available configuration options.

# Define the repositories to mirror.
mirror:
- url: https://github.com/org/repo1
  ref: refs/heads/master
  dir: repo1

- url: https://github.com/org/repo2
  ref: refs/heads/main
  dir: repo2

# Git configuration to be applied when mirror-tool creates commits.
# Any arbitrary config can be set, but this is most commonly needed
# just to set the name/email on merge commits.
git_config:
  user.name: "mirror-tool"
  user.email: "noreply@example.com"

# Message for generated commits.
# This is a Jinja template.
commitmsg: |-
  Merge {{commits[0].revision_abbrev}} to {{mirror.dir}}

  {{commits|length}} commit(s) are being merged.

  {% for commit in commits %}
  - {{ commit.revision_abbrev }} {{ commit.subject }}
  {%- endfor %}

# Configures the GitLab merge request integration.
gitlab_merge:
  # If enabled, the update command will create/update a GitLab
  # merge request whenever a mirrored repo is updated.
  enabled: true

  # Token used to authenticate with GitLab.
  #
  # Currently, this token must always be of the format '$SOME_VARIABLE',
  # and the token will be accessed from that environment variable at
  # runtime. If running from a GitLab CI/CD pipeline, this should be
  # set as a protected CI variable.
  token: $GITLAB_MIRROR_TOKEN

  # Source branch used for merge requests.
  # WARNING: update will do force pushes to this branch!
  src: latest

  # Target branch used for merge requests.
  # The following example supposes that the target branch is used
  # to perform some kind of deployment.
  dest: deploy

  # Title for the merge request.
  # This is a Jinja template.
  title: "Deploy changes [{{ datetime_day }}]"

  # Any desired labels to add onto the merge request.
  labels:
  - deploy

  # Description for the merge request.
  # This is a Jinja template.
  description: |-
    Automated update of dependencies generated by
    {{ env.CI_JOB_URL }}.

    Submitting this merge request will trigger a deployment.

  # Comment(s) to be added when a merge request is created or updated.
  # Can be used to ping reviewers.
  # If omitted, comments won't be added.
  # These are Jinja templates.
  comment:
    create: "@some-team: please review and submit."
    update: "@some-team: merge request has been updated, please re-review."

# Configures GitLab promotion between branches.
# A list of (src, dest) branch pairs with other config.
# Most config has the same meaning as in gitlab_merge.
gitlab_promote:
- src: stage
  dest: prod
  title: "Promote from stage to prod [{{datetime_day}}]"
  token: $GITLAB_MIRROR_TOKEN
  labels:
  - promote
  description: |-
    Automated promotion of {{ src_mr.web_url }} to prod.

Jinja context

Some configuration elements are described above as Jinja templates. The following variables are available for use within the templates:

env (dict)

• All environment variables at the time mirror-tool is invoked.
• If running in GitLab CI/CD, can be used to access the CI/CD variables.
• Example: {{ env.CI_JOB_URL }} => https://gitlab.example.com/someteam/somerepo/-/jobs/6366493

datetime_iso8601 (str)

• Current UTC date/time, in ISO8601 format, with seconds precision.
• Example: 2022-05-10T05:28:26Z

datetime_minute (str)

• Current UTC date/time, with minutes precision.
• Example: 2022-05-10 05:28

datetime_day (str)

• Current UTC date.
• Example: 2022-05-10

datetime_week (str)

• Current UTC year and week of year.
• Example: 2022wk19 for week 19 of 2022.

updates (list[UpdateInfo]) (update only)

In most Jinja contexts for the update and update-local commands, this is a list of objects of the following form:

UpdateInfo(
  mirror=Mirror(
    url="https://github.com/rohanpm/mirror-tool",
    ref="refs/heads/main",
    dir="mirror-tool"
  ),

  # Objects representing commits included in the update, starting
  # with the most recent.
  #
  # If there is a large number of commits being handled, some may be
  # elided from this list.
  commits=[
    Commit(
      revision="472b7797518b963f8ab381c39858c18b2b784c2e",
      revision_abbrev="472b779",
      author_name="Rohan McGovern",
      author_email="rohan@mcgovern.id.au",
      author_email_local="rohan",
      author_datetime=datetime.datetime(2022, 5, 26, 0, 24, 37),
      committer_name="Rohan McGovern",
      committer_email="rohan@mcgovern.id.au",
      committer_email_local="rohan",
      committer_datetime=datetime.datetime(2022, 5, 26, 0, 37, 37),
      subject="Raise test coverage to 100%",
      body="Do this, that and some other\nthings as well.",
      url="https://github.com/rohanpm/mirror-tool/commit/472b7797518b963f8ab381c39858c18b2b784c2e",
    ),
    ...,
  ],

  # Total number of commits in the update (may be more than len(commits)
  # if some were elided).
  commit_count=4,

  # Number of commits omitted from 'commits' object.
  # For example, if an update pulled 200 commits, only the most recent 20
  # may appear in 'commits', and this value will be set to 180.
  commit_elided_count=0,
)

In the Jinja context for commitmsg, as only a single update is being processed, updates is not defined. Instead, all of the fields shown above under UpdateInfo are directly included onto the context.

src_mr (dict) (promote only)

A merge request object which is now being promoted; i.e. a merge request previously created by mirror-tool and submitted to one branch, and now being promoted by mirror-tool to another branch.

The format of this object can be found in the GitLab API docs.

Only available for the promote command.

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.