Code ingress with git #611

ots22 · 2020-04-23T17:28:42Z

JR: This PR is out-of-date and includes several changes that are already incorporated into master. I've migrated the most important changes to 264-code-ingress-gitlab and will make a PR from there once tested.

The overall concept is shown in the diagram:

gitlab-ingress.pdf
Key:

arrow with solid head: connection initiated from tail to head
arrow with empty head: something else
box: a virtual machine
"document" symbol: a script - runs steps indicated by arrows of the same colour
"folder" symbol: a git repo
cylinder: data

An Admin (via a script) creates a snapshot of a single commit of the requested repo, and also creates a new repo on a GitLab instance (the "Reviewer" or "External" GitLab) that is only visible to Reviewers. Once two Reviewers approve, the snapshot is made available on the ("Internal") GitLab (the same as used by Users), under the user "Ingress."

The workflow is as follows:

User sends a request (out of band) for an external git resource to Admin (who should be authorized to clone it)
Admin can perform a preliminary check by themselves at this stage
Admin runs "SRE_Upload_Git_Repo_to_GitlabExternal.ps1", passing (as supplied by the requesting User):
- The SRE id
- The GitHub URL to requested resource
- The name of the repo to use internally (which may be different from the basename of the URL path to avoid name clashes, since all requested repos end up in a single namespace)
- The commit hash (to snapshot)
- The name of the branch to use internally
A Reviewer logs in to the External GitLab. They will be able to see two groups: "unapproved" and "approval", with a repo with the above name in each. There will be an open Merge Request from the repo in "unapproved" to the one in "approval" with the changes introduced by the most recent snapshot (effectively squashing all commits since the last approved snapshot).
The Reviewer reviews the code using the usual GitLab facilities, with an opportunity for discussion with the other Reviewer. They indicate their approval or disapproval using the "thumbs up" or "thumbs down" reaction to the Merge Request. (We do not have Merge Request approvals available on our version of GitLab).
The merge is made automatically (by a cron job), provided that there are:
- two "thumbs up" responses
- no "thumbs down" responses
- no unresolved comments
All "approval" repos are mirrored to the Internal GitLab, where they belong to the user "Ingress", and are visible alongside User repositories there.

Closes #257. Closes #264.

deployment/common/Configuration.psm1

deployment/secure_research_environment/setup/Setup_SRE_WebApp_Servers.ps1

deployment/common/Configuration.psm1

deployment/secure_research_environment/setup/Setup_SRE_WebApp_Servers.ps1

deployment/secure_research_environment/cloud_init/cloud-init-gitlab-external.template.yaml

…ngress, within cw20/Test A sandbox Progress during call on 2020-04-23 Co-authored-by: jack89roberts <jack89roberts@users.noreply.github.com> Co-authored-by: jemrobinson <jemrobinson@users.noreply.github.com> Co-authored-by: nbarlowATI <nbarlowATI@users.noreply.github.com>

Co-authored-by: jemrobinson <jemrobinson@users.noreply.github.com> Co-authored-by: nbarlowATI <nbarlowATI@users.noreply.github.com> Co-authored-by: ots22 <ots22@users.noreply.github.com>

Adds a user on gitlab internal that can be used to ingress repos from gitlab external: * Config and setup changed to add secrets for username, password and API token (stored in keyvault). * gitlab-rails commands in cloud init to add the user and generate the token. By default the username is "external" with email "external@<gitlab-domain-host>", e.g. external@cw20.dsgroupdev.co.uk. The email isn't valid/used but the gitlab server will only accept new users from the given domain.

Needs testing. Also renames some parameters and secrets to make the distinction between external and internal clearer.

…rnal Also add creation of "approved" and "unapproved" groups on gitlab external

For local gitlab testing on my laptop only - not a secret valid for anything in safe haven.

…load to GitlabExternal VM via blob storage

…find LDAP users)

…ment.template.ps1 idempotent by removing CAP/RAP settings

…ipt now

…ady exist

- Add rules for outbound connections identity server - Remove inbound RDP rule - Add inbound rule for session host

martintoreilly · 2020-07-16T10:01:52Z

deployment/common/Deployments.psm1

-        $null = Get-AzNetworkSecurityRuleConfig -Name $Name -NetworkSecurityGroup $NetworkSecurityGroup -ErrorVariable notExists -ErrorAction SilentlyContinue
+        $rule = Get-AzNetworkSecurityRuleConfig -Name $Name -NetworkSecurityGroup $NetworkSecurityGroup -ErrorVariable notExists -ErrorAction SilentlyContinue


Didn't we say we were going to put the refactor of this and related functions into it's own PR?

Yes, sorry, this commit probably shouldn't have ended up on this branch - I'll make a separate PR for this.

The absence of this rule was blocking RDP connections between the APP session host and the DSVM, since this NSG is applied to the DATA subnet. There is almost certainly something more fundamentally wrong, but this patch means that if this branch is deployed (e.g. in a test environment), it won't cause this issue.

jemrobinson · 2021-01-11T13:53:33Z

Closing this PR as it is out-of-date and includes several changes that are already incorporated into master. I've migrated the most important changes to 264-code-ingress-gitlab and will make a PR from there once tested.

ots22 mentioned this pull request Apr 23, 2020

Define Git-based code ingress #264

Closed

martintoreilly reviewed Apr 28, 2020

View reviewed changes

deployment/common/Configuration.psm1 Outdated Show resolved Hide resolved

martintoreilly reviewed Apr 28, 2020

View reviewed changes

deployment/secure_research_environment/setup/Setup_SRE_WebApp_Servers.ps1 Outdated Show resolved Hide resolved

martintoreilly reviewed Apr 28, 2020

View reviewed changes

deployment/common/Configuration.psm1 Show resolved Hide resolved

martintoreilly reviewed Apr 28, 2020

View reviewed changes

deployment/secure_research_environment/setup/Setup_SRE_WebApp_Servers.ps1 Outdated Show resolved Hide resolved

jack89roberts reviewed May 12, 2020

View reviewed changes

deployment/secure_research_environment/cloud_init/cloud-init-gitlab-external.template.yaml Outdated Show resolved Hide resolved

jack89roberts reviewed May 12, 2020

View reviewed changes

deployment/secure_research_environment/cloud_init/cloud-init-gitlab-external.template.yaml Outdated Show resolved Hide resolved

ots22 and others added 23 commits May 14, 2020 21:06

Add a basic cloud init template for gitlab external

3c0aecd

Co-authored-by: jemrobinson <jemrobinson@users.noreply.github.com> Co-authored-by: nbarlowATI <nbarlowATI@users.noreply.github.com> Co-authored-by: ots22 <ots22@users.noreply.github.com>

Remove debugging print statement and unused variable

6056af1

add network security group rule (InboundDenyAll) for nsgGitlabExternal

8b8d20c

Add a new session server for holding connections to review boxes

424bbd3

Add SSH key and git setup on gitlab external

3f04679

WIP Script to update GitLab projects from whitelist

ff48810

Updated review session server settings

8507b63

Updated RDS deployment configuration

7c8d377

Add gitlab server to gitlab external

9a52480

Needs testing. Also renames some parameters and secrets to make the distinction between external and internal clearer.

Updated SRE teardown script

b9d5788

Removed cw20 configs as we are using testa now

f8c415a

WIP script for monitoring and accepting merge requests on gitlab exte…

ccdc93f

…rnal Also add creation of "approved" and "unapproved" groups on gitlab external

Merge requests script: Replace print statements with logging

fd62e1f

Remove localhost token accidentally added in previous commit

f139b2f

For local gitlab testing on my laptop only - not a secret valid for anything in safe haven.

Changed name of test researcher

69df12d

Updated RDS scripts

49df919

wait for gitlab server health check before interacting with api

23a2733

replace bash syntax

59d3073

Script to create zipfile from specified commit on a git repo, then up…

ab9de4c

…load to GitlabExternal VM via blob storage

couple more shell formatting fixes

0881963

Add function to create or push to a repo on GITLAB-INTERNAL

b5b2337

ots22 and others added 24 commits July 3, 2020 14:38

Add missing gitlab-rb-host template substitution (fixes inability to …

67baafa

…find LDAP users)

containerName -> tmpContainerName

cae41f3

Fix cleaning up resources

7b844c3

Fix flag to Remove-AzStorageContainer

815f2f0

Merge branch 'master' into 264-gitlab-ingress

e832a0b

Merge branch 'master' into 264-gitlab-ingress

86bba46

Minor fix to blob storage cleaner

0018a69

RDS fixes from redeploying. Includes a fix to make Deploy_RDS_Environ…

243980d

…ment.template.ps1 idempotent by removing CAP/RAP settings

Removed wait for cloud-init as this is included in the deployment scr…

59b5d08

…ipt now

Updates from webapps redeploy

6d36d72

Pass subprocess.run args as a list

a80cb82

Use correct LDAP OU for research users on webapp VMs

48d2be3

Make the GitLab service user the owner of the zipfiles

2f505d2

Fix: string interpolation

8041233

Recover from partially-completed run: ensure cloned repo doesn't alre…

a77afc9

…ady exist

Fix: argument order

b7ca0da

Lint

8d477fc

Fix typo: gitlabreview -> gitlabReview

6173b11

Check for correct (created) status code

bb6ca85

Inline clone, commit and push

cc4977b

gitlabUsername -> gitlabUserIngressUsername for cloud-init substitution

2ab4d54

Partial fix to NSG rules

678be5b

Rename NSG rules template; adjust rules

6e7d2a1

- Add rules for outbound connections identity server - Remove inbound RDP rule - Add inbound rule for session host

Return PSNetworkSecurityGroup object from Add-NetworkSecurityGroupRule

3462ba6

martintoreilly reviewed Jul 16, 2020

View reviewed changes

jemrobinson closed this Jan 11, 2021

edwardchalstrey1 mentioned this pull request May 11, 2021

Reopen code ingress with git #1000

Closed

jemrobinson deleted the 264-gitlab-ingress branch September 27, 2022 13:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Code ingress with git #611

Code ingress with git #611

ots22 commented Apr 23, 2020 •

edited by jemrobinson

Loading

martintoreilly Jul 16, 2020

ots22 Jul 16, 2020

jemrobinson commented Jan 11, 2021

		$null = Get-AzNetworkSecurityRuleConfig -Name $Name -NetworkSecurityGroup $NetworkSecurityGroup -ErrorVariable notExists -ErrorAction SilentlyContinue
		$rule = Get-AzNetworkSecurityRuleConfig -Name $Name -NetworkSecurityGroup $NetworkSecurityGroup -ErrorVariable notExists -ErrorAction SilentlyContinue

Code ingress with git #611

Code ingress with git #611

Conversation

ots22 commented Apr 23, 2020 • edited by jemrobinson Loading

martintoreilly Jul 16, 2020

Choose a reason for hiding this comment

ots22 Jul 16, 2020

Choose a reason for hiding this comment

jemrobinson commented Jan 11, 2021

ots22 commented Apr 23, 2020 •

edited by jemrobinson

Loading