Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@main
- uses: actions/setup-python@v1
- uses: actions/setup-python@v4
with:
python-version: "3.7"
python-version: "3.12"
- name: Build docs
run: |
pip install pipenv
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@main
- uses: actions/setup-python@v1
- uses: actions/setup-python@v4
with:
python-version: '3.7'
python-version: '3.12'
- name: Build docs
run: |
pip install pipenv
Expand Down
2 changes: 1 addition & 1 deletion Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,4 @@ sphinx = "*"
recommonmark = "*"

[requires]
python_version = "3.7"
python_version = "3.12"
368 changes: 241 additions & 127 deletions Pipfile.lock

Large diffs are not rendered by default.

49 changes: 22 additions & 27 deletions basics/github.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,15 @@
# GitHub

## GitHub Organization Security

As of 2025, the GitHub Organization [MITLibraries](https://github.com/MITLibraries) has updated security settings in place.

There are only two "owners" of the GitHub Organization: the team lead for EngX and the team lead for InfraEng. All other user accounts in the GitHub Organization are "members."

Additionally, all of the repositories have been updated to ensure that the right teams have the right access to the right repositories. In general, this means that infrastructure repositories have the [InfraEng](https://github.com/orgs/MITLibraries/teams/infraeng) team with the built-in **maintain** Role, the data engineering repositories have the [DataEng](https://github.com/orgs/MITLibraries/teams/dataeng) team with the built-in **maintain** Role, and the discovery/ui/web repositories have the [EngX](https://github.com/orgs/MITLibraries/teams/engx) team with the built-in **maintain** Role.

Member accounts **do** have the right to create new repositories. Once they create a repository and configure it with the [Recommended repo settings](#recommended-repo-settings), they should notify at least one of the Organization owners who will then set up the rest of the collaboration permissions.

## Repo naming

Naming is important (see [There's Power in Naming ...](https://www.taketheleadwomen.com/blog/theres-power-in-naming-and-power-in-knowing-your-name)). More specific to our work, there are a few guidelines for naming new repos
Expand Down Expand Up @@ -30,7 +40,7 @@ In most cases, [GitHub flow](https://guides.github.com/introduction/flow/) will

Ideally, feature branches should encompass small, targeted changes. Keeping your work focused will mean fewer merge conflicts. When building a new code base this may not always be possible, but what you want to try and avoid are long running branches. By the time you want to merge, the distance between your feature branch and main will usually mean a lot of conflict resolution in your future.

There is an exception to this for Infrastructure code. For the Terraform code, InfraEng uses a Simplified Git Flow model (which could also be seen as GitHub Flow on steroids). In this case, each deployment environment (prod & stage) is linked to a long-running branch in the repository. As in GitHub Flow, feature branches should encompass small, targeted changes. They are always branched from and merged to the `stage` branch. And the only merges to the `main` branch come from approved merges on the `stage` branch. More details on this model can be found at [Simplified Git Flow](https://medium.com/goodtogoat/simplified-git-flow-5dc37ba76ea8) and at [Simplified Git Flow for InfraEng](https://mitlibraries.atlassian.net/wiki/spaces/IN/pages/2920480769/Simplified+Git+Flow+For+InfraEng).
There is an exception to this for Infrastructure code. For the Terraform code, InfraEng uses a Simplified Git Flow model. In this case, each deployment environment (dev, stage, & prod) is linked to a long-running branch in the repository. The `dev` branch is the working branch for all changes/features. PRs and code reviews are required for any merges to `stage` and the only merges to the `main` branch come from approved merges on the `stage` branch. More details on this model can be found at [Simplified Git Flow](https://medium.com/goodtogoat/simplified-git-flow-5dc37ba76ea8) and at [Simplified Git Flow for InfraEng](https://mitlibraries.atlassian.net/wiki/spaces/IN/pages/2920480769).

## Merging and Rebasing

Expand All @@ -50,9 +60,7 @@ $ git merge <branch-name> --no-ff

## Pull Requests

We use pull requests to merge changes to a codebase. In nearly all instances, pull requests should be reviewed by
another person before merging. Check out our [code review guide](/collaboration/code_review.html) for more information
on how this works.
We use pull requests to merge changes to a codebase. In nearly all instances, pull requests should be reviewed by another person before merging. Check out our [code review guide](/collaboration/code_review.html) for more information on how this works.

## Github and JIRA

Expand All @@ -62,38 +70,25 @@ In general, if you are creating code on GitHub, commenting and review should hap

## GitHub and Code Climate

We use [Code Climate](https://codeclimate.com/)'s GitHub integration to monitor
for code smells. Code Climate is free for open source repositories.
We use [Code Climate](https://codeclimate.com/)'s GitHub integration to monitor for code smells. Code Climate is free for open source repositories.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is a broad truth in our org anymore and only EngX seems to be using it. It's also gotten really weird in the past few weeks as they are rolling out a replacement product.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll leave this for another commit by you or someone else.


To connect Code Climate to your repo:
1. [Log in to Code Climate](https://codeclimate.com/login) with your GitHub
account
2. If it's your first time logging in, grant Code Climate OAuth permissions as
prompted

1. [Log in to Code Climate](https://codeclimate.com/login) with your GitHub account
2. If it's your first time logging in, grant Code Climate OAuth permissions as prompted
3. Follow the prompts to add your repo:
* On the dashboard, click the 'Open source' link, then click 'Add a repository'
* Find your repo on the ensuing list, then click 'Add repository' _(note: you
may need to click the 'Sync now' link for all repos to appear)_
4. Continue to your repo dashboard in Code Climate, then click the 'Repo settings'
tab. Click 'GitHub' on the menu, then install 'Pull request status updates' and
'Webhook on GitHub'
5. Code Climate should now run in GitHub every time you open a PR. In your
GitHub repo, you can confirm this by going to Settings -> Webhooks and make
sure the Code Climate webhook is listed _(note: if Code Climate is stuck pending
in your PR, it might be because you opened the PR prior to configuring Code Climate)_
* Find your repo on the ensuing list, then click 'Add repository' _(note: you may need to click the 'Sync now' link for all repos to appear)_
4. Continue to your repo dashboard in Code Climate, then click the 'Repo settings' tab. Click 'GitHub' on the menu, then install 'Pull request status updates' and 'Webhook on GitHub'
5. Code Climate should now run in GitHub every time you open a PR. In your GitHub repo, you can confirm this by going to Settings -> Webhooks and make sure the Code Climate webhook is listed _(note: if Code Climate is stuck pending in your PR, it might be because you opened the PR prior to configuring Code Climate)_

## Updating the default branch in GitHub

While all new repos will be created with `main` as the default branch, older
repos may still use `master` and should be migrated. GitHub now provides an
easy way to do this in the web UI. In the repo's settings, you can
change the name of the default branch.
While all new repos will be created with `main` as the default branch, older repos may still use `master` and should be migrated. GitHub now provides an easy way to do this in the web UI. In the repo's settings, you can change the name of the default branch.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

older repos may still use `master` and should be migrated if possible

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll leave this for another commit by you or someone else.


Once you change the branch name, you and your collaborators will need to pull
the changes to your cloned repos.
Once you change the branch name, you and your collaborators will need to pull the changes to your cloned repos.

Additionally, you should check any automations you have configured to ensure
they are watching the new branch (e.g., CodeClimate, CI/CD).
Additionally, you should check any automations you have configured to ensure they are watching the new branch (e.g., CodeClimate, CI/CD).

- - -

Expand Down
3 changes: 0 additions & 3 deletions deploy/airflow.md

This file was deleted.

2 changes: 1 addition & 1 deletion deploy/aws-2.0.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# AWS Organization

As of December 2021, MIT Libraries has shifted to an AWS Organization containing multiple AWS Accounts. We have enabled AWS SSO, integrated with Touchstone, for access to all of our AWS Accounts in our AWS Organization.
As of December 2021, MIT Libraries has shifted to an AWS Organization containing multiple AWS Accounts. We have enabled AWS SSO, integrated with Touchstone/Okta, for access to all of our AWS Accounts in our AWS Organization.

## Getting Started

Expand Down
18 changes: 9 additions & 9 deletions deploy/terraform-1.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,23 @@ This contains most of the general documenation for how we use Terraform at MIT L

## Getting Started

All new Terraform work should be using version 1.0.x. We use Terraform Cloud for both remote state and remote execution. This means that for all future Terraform work, you'll need to configure your GitHub access, your AWS Access (see [AWS 2.0](/deploy/aws-2.0)), your Terraform Cloud access (see **Terraform Cloud** section below), and install the latest version of the Terraform 1.0 CLI (see **Terraform CLI (local)** section below).
All new Terraform work should be using version 1.x.y (where x > 9). We use Terraform Cloud for both remote state and remote execution. This means that for all future Terraform work, you'll need to configure your GitHub access, your AWS Access (see [AWS 2.0](/deploy/aws-2.0)), your Terraform Cloud access (see **Terraform Cloud** section below), and install the latest version of the Terraform 1.0 CLI (see **Terraform CLI (local)** section below).

## GitHub

As part of your onboarding, you created a GitHub account and were invited to join the MIT Libraries GitHub Organization. Your initial privileges in GitHub are limited until you have completed the onboarding process.

Our Terraform code is spread across multiple, targeted repositories, generally one Terraform repository per application with a few Terraform repositories that manage the core infrastructure in our AWS Accounts. The [mitlib-tfc-mgmt][https://github.com/MITLibraries/mitlib-tfc-mgmt] Terraform repository manages all the other Terraform repositories (_one ring to rule them all?_). If you need an additional Terraform repository for new application-specific infrastructure, there is a process to add the necessary information to the management repository to create the necessary bits in GitHub (and Terraform Cloud).
Our Terraform code is spread across multiple, targeted repositories, generally one Terraform repository per application with a few Terraform repositories that manage the core infrastructure in our AWS Accounts. The [mitlib-tfc-mgmt](https://github.com/MITLibraries/mitlib-tfc-mgmt) Terraform repository manages all the other Terraform repositories (_one ring to rule them all?_). If you need an additional Terraform repository for new application-specific infrastructure, there is a process to add the necessary information to the management repository to create the necessary bits in GitHub (and Terraform Cloud).

## Terraform Cloud

There is a single [Terraform Cloud Organization for MIT Libraries](https://app.terraform.io/app/MITLibraries). As part of your onboarding, your kerb was linked to this organization. You will need to configure access to that organization for your workstation so that you can run speculative plans for your Terraform code. See the [Terraform Cloud CLI Login](https://learn.hashicorp.com/tutorials/terraform/cloud-login?in=terraform/0-13) Getting Started article from HashiCorp for a walkthrough and see the [CLI Configuration](https://www.terraform.io/docs/cli/config/config-file.html) page in the full Terraform Cloud documentation site for details.
There is a single [Terraform Cloud Organization for MIT Libraries](https://app.terraform.io/app/MITLibraries). As part of your onboarding, your kerb was linked to this organization. You will need to configure access to that organization for your workstation so that you can run speculative plans for your Terraform code. See the [Terraform Cloud CLI](https://developer.hashicorp.com/terraform/tutorials/cli) page from HashiCorp for a walkthrough and see the [CLI Configuration](https://developer.hashicorp.com/terraform/cli/config/config-file) page in the full Terraform Cloud documentation site for details.

All of our Terraform repositories are linked to Terraform Cloud workspaces (this is all managed by the repository referenced in the preceding section). Generally, once the Terraform repository is created and the Terraform Cloud workspace is created and linked, there are no other changes needed to the Terraform Cloud workspace save one: it is required that you manually trigger the first `terraform plan` through the Terraform Cloud workspace console. For more details on how we are using Terraform Cloud, see the [Terraform Cloud and GitHub](https://mitlibraries.atlassian.net/wiki/spaces/IN/pages/2893381658/USE%3A+Terraform+Cloud+and+GitHub) article in the InfraEng secure KB.
All of our Terraform repositories are linked to Terraform Cloud workspaces (this is all managed by the repository referenced in the preceding section). Generally, once the Terraform repository is created and the Terraform Cloud workspace is created and linked, there are no other changes needed to the Terraform Cloud workspace save one: it is required that you manually trigger the first `terraform plan` through the Terraform Cloud workspace console. For more details on how we are using Terraform Cloud, see the [Terraform Cloud and GitHub](https://mitlibraries.atlassian.net/wiki/x/GoB1r) article in the InfraEng secure KB.

## Terraform CLI (local)

Even though we are using Terraform Cloud, you will still need a local copy of the Terraform CLI. We are currently building Terraform code compatible with Terraform 1.0. You will likely need older versions of the Terraform CLI to manage some of our legacy Terraform code. There are multiple tools for managing your Terraform CLI, and no one really cares which one you use. As long as you can fully control your version of Terraform and easily switch between versions.
Even though we are using Terraform Cloud, you will still need a local copy of the Terraform CLI. We are currently building Terraform code compatible with Terraform 1.0. There are multiple tools for managing your Terraform CLI, and no one really cares which one you use. As long as you can fully control your version of Terraform and easily switch between versions.

### Version Managers for Terraform

Expand All @@ -38,10 +38,10 @@ The managed Terraform repositories and Terraform Cloud workspaces have a pre-con

## Repositories in detail

The Terraform 1.0 repositories are private by default. Any Terraform repository can be made public by request, but this will trigger a detailed security review before the visibility settings can be changed. Please see [this InfraEng Decision document](https://mitlibraries.atlassian.net/wiki/spaces/IN/pages/2907865138).
The Terraform 1.x repositories are private by default. Any Terraform repository can be made public by request, but this will trigger a detailed security review before the visibility settings can be changed. Please see [this InfraEng Decision document](https://mitlibraries.atlassian.net/wiki/spaces/IN/pages/2907865138).

The Terraform 1.0 repositories are targeted: one repo per application infrastructure. This is documented in [this Decision document](https://mitlibraries.atlassian.net/wiki/spaces/IN/pages/2899935256/Decision+004%3A+Use+small+repos+for+Terraform+code+instead+of+a+mono-repo).
The Terraform 1.x repositories are targeted: one repo per application infrastructure. This is documented in [this Decision document](https://mitlibraries.atlassian.net/wiki/x/GIDZr).

The Terraform 1.0 repositories serve multiple deployment environments: one repo builds both stage and prod infrastructure. See this [Decision document](https://mitlibraries.atlassian.net/wiki/spaces/IN/pages/2901278721) as well as [this one](https://mitlibraries.atlassian.net/wiki/spaces/IN/pages/2900656132/Decision+010%3A+Core+Infrastructure+repos+will+have+three+long-running+branches%3A+dev%2C+stage%2C+and+main).
The Terraform 1.x repositories serve multiple deployment environments: one repo builds both stage and prod infrastructure. See this [Decision document](https://mitlibraries.atlassian.net/wiki/x/BIDkr).

The Terraform 1.0 repositories are managed with a simplified version of Git Flow (instead of GitHub Flow). Details of how this will work is documented in [Simplified Git Flow for InfraEng](https://mitlibraries.atlassian.net/wiki/spaces/IN/pages/2920480769/Simplified+Git+Flow+For+InfraEng). The source of this model can be found in the [Simplified Git Flow](https://medium.com/goodtogoat/simplified-git-flow-5dc37ba76ea8) post on [medium.com](https://medium.com).
The Terraform 1.x repositories are managed with a simplified version of Git Flow (instead of GitHub Flow). Details of how this will work is documented in [Simplified Git Flow for InfraEng](https://mitlibraries.atlassian.net/wiki/x/AQATrg). The source of this model can be found in the [Simplified Git Flow](https://medium.com/goodtogoat/simplified-git-flow-5dc37ba76ea8) post on [medium.com](https://medium.com).
Loading