Skip to content

Commit

Permalink
Merge pull request #103 from Erotemic/dev/0.15.9
Browse files Browse the repository at this point in the history 
Start branch for 0.15.9
  • Loading branch information
@Erotemic
Erotemic committed Sep 24, 2021
2 parents 8de07a2 + 135e379 commit 5aa4fc5
Show file tree
Hide file tree
Showing 32 changed files with 1,173 additions and 588 deletions.
303 changes: 1 addition & 302 deletions .circleci/config.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,191 +19,7 @@ __doc__: &__doc__
- |
__doc__="""
============
CIRCLECI INSTRUCTIONS
============
This file was designed to be used as a template. You can adapt it to
new projects with a few simple changes. Namely perform the following
search and replaces.
```bash
cat .circleci/config.yml | \
sed 's|GITHUB_USER|Erotemic|g' | \
sed 's|PYPKG|xdoctest|g' | \
sed 's|GPG_ID|travis-ci-Erotemic|g' | \
sed 's|CIRCLE_CI_SECRET|CIRCLE_CI_SECRET|g' | \
tee /tmp/repl && colordiff .circleci/config.yml /tmp/repl
# overwrite if you like the diff
cp /tmp/repl .circleci/config.yml
```
To use this script you need the following configurations on your
CircleCI account.
NOTES
-----
* This script will require matainence for new releases of Python
CIRCLECI SECRETS # NOQA
--------------
Almost all of the stages in this pipeline can be performed on a local
machine (making it much easier to debug) as well as the circleci-ci
machine. However, there are a handeful of required environment
variables which will contain sensitive information. These variables are
* TWINE_USERNAME - this is your pypi username
twine info is only needed if you want to automatically publish to pypi
* TWINE_PASSWORD - this is your pypi password
* CIRCLE_CI_SECRET - We will use this as a secret key to encrypt/decrypt gpg secrets
This is only needed if you want to automatically sign published
wheels with a gpg key.
* PERSONAL_GITHUB_PUSH_TOKEN -
This is only needed if you want to automatically git-tag release branches.
To make a API token go to:
https://docs.github.com/en/free-pro-team@latest/github/authenticating-to-github/creating-a-personal-access-token
Instructions:
Browse to:
https://app.circleci.com/settings/project/github/Erotemic/xdoctest/environment-variables
Do whatever you need to locally access the values of these variables
echo $TWINE_USERNAME
echo $PERSONAL_GITHUB_PUSH_TOKEN
echo $CIRCLE_CI_SECRET
echo $TWINE_PASSWORD
For each one, click "Add Environment Variable" and enter the name
and value. Unfortunately this is a manual process.
WARNING: Ensure that your CircleCI project settings do not allow Forks
to view environment variables.
TODO: Can you protect branches on CircleCI, is that the default?
- ANS: CircleCI provides contexts: https://circleci.com/docs/2.0/contexts/
TODO: Look into secrethub
WARNING: If an untrusted actor gains the ability to write to a
protected branch, then they will be able to exfiltrate your secrets.
WARNING: These variables contain secret information. Ensure that these
the protected and masked settings are enabled when you create them.
ENCRYPTING GPG SECRETS
----------------------
The following script demonstrates how to securely encrypt a secret GPG key. It is assumed that you have
a file secret_loader.sh that looks like this
```bash
source secretfile
```
and then a secretfile that looks like this
```bash
#!/bin/bash
echo /some/secret/file
export TWINE_USERNAME=<pypi-username>
export TWINE_PASSWORD=<pypi-password>
export CIRCLE_CI_SECRET="<a-very-long-secret-string>"
export PERSONAL_GITHUB_PUSH_TOKEN='git-push-token:<token-password>'
```
You should also make a secret_unloader.sh that points to a script that
unloads these secret variables from the environment.
Given this file-structure setup, you can then run the following
commands verbatim. Alternatively just populate the environment
variables and run line-by-line without creating the secret
loader/unloader scripts.
```bash
# THIS IS NOT EXECUTE ON THE CI, THIS IS FOR DEVELOPER REFERENCE
# ON HOW THE ENCRYPTED GPG KEYS ARE SETUP.
# Load or generate secrets
source $(secret_loader.sh)
echo $CIRCLE_CI_SECRET
echo $TWINE_USERNAME
# ADD RELEVANT VARIABLES TO CIRCLECI SECRET VARIABLES
# https://app.circleci.com/settings/project/github/Erotemic/xdoctest/environment-variables
# See previous CIRCLE_CI section for more details
# HOW TO ENCRYPT YOUR SECRET GPG KEY
IDENTIFIER="travis-ci-Erotemic"
GPG_KEYID=$(gpg --list-keys --keyid-format LONG "$IDENTIFIER" | head -n 2 | tail -n 1 | awk '{print $1}' | tail -c 9)
echo "GPG_KEYID = $GPG_KEYID"
# Export plaintext gpg public keys, private keys, and trust info
mkdir -p dev
gpg --armor --export-secret-keys $GPG_KEYID > dev/ci_secret_gpg_key.pgp
gpg --armor --export $GPG_KEYID > dev/ci_public_gpg_key.pgp
gpg --export-ownertrust > dev/gpg_owner_trust
# Encrypt gpg keys and trust with CI secret
GLKWS=$CIRCLE_CI_SECRET openssl enc -aes-256-cbc -pbkdf2 -md SHA512 -pass env:GLKWS -e -a -in dev/ci_public_gpg_key.pgp > dev/ci_public_gpg_key.pgp.enc
GLKWS=$CIRCLE_CI_SECRET openssl enc -aes-256-cbc -pbkdf2 -md SHA512 -pass env:GLKWS -e -a -in dev/ci_secret_gpg_key.pgp > dev/ci_secret_gpg_key.pgp.enc
GLKWS=$CIRCLE_CI_SECRET openssl enc -aes-256-cbc -pbkdf2 -md SHA512 -pass env:GLKWS -e -a -in dev/gpg_owner_trust > dev/gpg_owner_trust.enc
echo $GPG_KEYID > dev/public_gpg_key
# Test decrpyt
cat dev/public_gpg_key
GLKWS=$CIRCLE_CI_SECRET openssl enc -aes-256-cbc -pbkdf2 -md SHA512 -pass env:GLKWS -d -a -in dev/ci_public_gpg_key.pgp.enc
GLKWS=$CIRCLE_CI_SECRET openssl enc -aes-256-cbc -pbkdf2 -md SHA512 -pass env:GLKWS -d -a -in dev/gpg_owner_trust.enc
GLKWS=$CIRCLE_CI_SECRET openssl enc -aes-256-cbc -pbkdf2 -md SHA512 -pass env:GLKWS -d -a -in dev/ci_secret_gpg_key.pgp.enc
source $(secret_unloader.sh)
# Look at what we did, clean up, and add it to git
ls dev/*.enc
rm dev/gpg_owner_trust dev/*.pgp
git status
git add dev/*.enc
git add dev/public_gpg_key
```
TEST PERSONAL_GITHUB_PUSH_TOKEN
-------------------
The following script tests if your PERSONAL_GITHUB_PUSH_TOKEN environment variable is correctly setup.
```bash
docker run -it ubuntu
apt update -y && apt install git -y
git clone https://github.com/Erotemic/xdoctest.git
cd xdoctest
# do sed twice to handle the case of https clone with and without a read token
git config user.email "ci@circleci.com"
git config user.name "CircleCI-User"
URL_HOST=$(git remote get-url origin | sed -e 's|https\?://.*@||g' | sed -e 's|https\?://||g')
echo "URL_HOST = $URL_HOST"
git tag "test-tag4"
git push --tags "https://${PERSONAL_GITHUB_PUSH_TOKEN}@${URL_HOST}"
# Cleanup after you verify the tags shows up on the remote
git push --delete origin test-tag4
git tag --delete test-tag4
```
Main CI has moved to github actions
""" # " # hack for vim yml syntax highlighter
version: 2
Expand Down Expand Up @@ -270,20 +86,6 @@ workflows:
filters:
<<: *__ignore_release__

- gpgsign/cp38-38-linux:
filters:
branches:
only:
- master
- release

- deploy/cp38-38-linux:
requires:
- gpgsign/cp38-38-linux
filters:
branches:
only:
- release

jobs:

Expand Down Expand Up @@ -380,85 +182,6 @@ jobs:
path: .coverage
destination: .coverage

.gpgsign_template: &gpgsign_template
<<:
- *common_template
steps:
- checkout
- run:
name: gpg_sign_dist
command: |
$PYTHON_EXE -m venv venv || virtualenv -v venv # first command is python3 || second is python2
. venv/bin/activate
export GPG_EXECUTABLE=gpg
export GPG_KEYID=$(cat dev/public_gpg_key)
echo "GPG_KEYID = $GPG_KEYID"
echo "-- QUERY GPG VERSION..."
$GPG_EXECUTABLE --version
echo "-- QUERY OPENSSL VERSION..."
openssl version
echo "-- QUERY GPG KEYS (done twice as a workaround)"
$GPG_EXECUTABLE --list-keys || $GPG_EXECUTABLE --list-keys
echo "-- QUERY GPG KEYS (done twice as a workaround)"
$GPG_EXECUTABLE --list-keys || $GPG_EXECUTABLE --list-keys
echo "-- Decrypt and import GPG Keys / trust"
# note CIRCLE_CI_SECRET is a protected variables only available on master and release branch
echo "CIRCLE_CI_SECRET = $CIRCLE_CI_SECRET"
GLKWS=$CIRCLE_CI_SECRET openssl enc -aes-256-cbc -pbkdf2 -md SHA512 -pass env:GLKWS -d -a -in dev/ci_public_gpg_key.pgp.enc | $GPG_EXECUTABLE --import
GLKWS=$CIRCLE_CI_SECRET openssl enc -aes-256-cbc -pbkdf2 -md SHA512 -pass env:GLKWS -d -a -in dev/gpg_owner_trust.enc | $GPG_EXECUTABLE --import-ownertrust
GLKWS=$CIRCLE_CI_SECRET openssl enc -aes-256-cbc -pbkdf2 -md SHA512 -pass env:GLKWS -d -a -in dev/ci_secret_gpg_key.pgp.enc | $GPG_EXECUTABLE --import
$GPG_EXECUTABLE --list-keys || echo "first one fails for some reason"
$GPG_EXECUTABLE --list-keys
# The publish script only builds wheels and does gpg signing if DO_UPLOAD is no
pip install requests[security] twine wheel
echo "Execute the publish script in dry mode"
MB_PYTHON_TAG=$MB_PYTHON_TAG DO_GPG=True GPG_KEYID=$GPG_KEYID TWINE_PASSWORD=$TWINE_PASSWORD TWINE_USERNAME=$TWINE_USERNAME GPG_EXECUTABLE=$GPG_EXECUTABLE CURRENT_BRANCH=release DEPLOY_BRANCH=release DO_UPLOAD=False DO_TAG=False ./publish.sh
- store_artifacts:
path: dist
destination: dist


.deploy_template: &deploy_template
<<:
- *common_template
steps:
- checkout
- run:
name: deploy
command: |
$PYTHON_EXE -m venv venv || virtualenv -v venv # first command is python3 || second is python2
. venv/bin/activate
export GPG_EXECUTABLE=gpg
export GPG_KEYID=$(cat dev/public_gpg_key)
# Decrypt and import GPG Keys / trust
# note CIRCLE_CI_SECRET is a protected variables only available on master and release branch
GLKWS=$CIRCLE_CI_SECRET openssl enc -aes-256-cbc -pbkdf2 -md SHA512 -pass env:GLKWS -d -a -in dev/ci_public_gpg_key.pgp.enc | $GPG_EXECUTABLE --import
GLKWS=$CIRCLE_CI_SECRET openssl enc -aes-256-cbc -pbkdf2 -md SHA512 -pass env:GLKWS -d -a -in dev/gpg_owner_trust.enc | $GPG_EXECUTABLE --import-ownertrust
GLKWS=$CIRCLE_CI_SECRET openssl enc -aes-256-cbc -pbkdf2 -md SHA512 -pass env:GLKWS -d -a -in dev/ci_secret_gpg_key.pgp.enc | $GPG_EXECUTABLE --import
$GPG_EXECUTABLE --list-keys || echo "first one fails for some reason"
$GPG_EXECUTABLE --list-keys
# Install twine
pip install six pyopenssl ndg-httpsclient pyasn1 -U
pip install requests[security] twine wheel
# Execute the publish script for real this time
MB_PYTHON_TAG=$MB_PYTHON_TAG DO_GPG=True GPG_KEYID=$GPG_KEYID TWINE_PASSWORD=$TWINE_PASSWORD TWINE_USERNAME=$TWINE_USERNAME GPG_EXECUTABLE=$GPG_EXECUTABLE CURRENT_BRANCH=release DEPLOY_BRANCH=release DO_UPLOAD=True DO_TAG=False ./publish.sh
# Have the server git-tag the release and push the tags
VERSION=$($PYTHON_EXE -c "import setup; print(setup.VERSION)")
# do sed twice to handle the case of https clone with and without a read token
URL_HOST=$(git remote get-url origin | sed -e 's|https\?://.*@||g' | sed -e 's|https\?://||g' | sed -e 's|git@||g' | sed -e 's|:|/|g')
echo "URL_HOST = $URL_HOST"
# A git config user name and email is required. Set if needed.
if [[ "$(git config user.email)" == "" ]]; then
git config user.email "ci@circleci.com"
git config user.name "CircleCI"
fi
if [ $(git tag -l "$VERSION") ]; then
echo "Tag already exists"
else
git tag $VERSION -m "tarball tag $VERSION"
git push --tags "https://${PERSONAL_GITHUB_PUSH_TOKEN}@${URL_HOST}"
fi

###################################
### INHERIT FROM BASE TEMPLATES ###
Expand Down Expand Up @@ -568,31 +291,9 @@ jobs:
- PYTHON_EXE: pypy3


# -- gpgsign + deploy

gpgsign/cp38-38-linux:
<<: *gpgsign_template
docker:
- image: circleci/python:3.8


deploy/cp38-38-linux:
<<: *deploy_template
docker:
- image: circleci/python:3.8


.__doc__: &__doc__
- |
# Test GPG sign works
load_secrets
circleci local execute \
-e CIRCLE_CI_SECRET=$CIRCLE_CI_SECRET \
--job gpgsign/cp38-38-linux
IMAGE_NAME=circleci/python:3.9
docker pull $IMAGE_NAME
Expand Down Expand Up @@ -669,8 +370,6 @@ jobs:
JOB_NAME=test_minimal/pypy3
circleci local execute --job $JOB_NAME
circleci local execute --job gpgsign/cp38-38-linux
JOB_NAME=test_full/pypy3
circleci local execute --job $JOB_NAME
Expand Down
8 changes: 8 additions & 0 deletions .github/dependabot.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
version: 2
updates:
# Maintain dependencies for GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
day: "friday"
Loading

0 comments on commit 5aa4fc5

Please sign in to comment.