Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/dp ssh concourse tags #129

Merged
merged 25 commits into from
Mar 6, 2023
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
92c59f7
env changes, ssh by instance id
gedge Apr 8, 2022
4e00c49
Merge branch 'main' into feature/dp-ssh-concourse-tags
gedge Apr 21, 2022
ddf93be
show instance IDs and as options for `dp ssh $env`
gedge Apr 21, 2022
ba368e4
add instance ID to 'dp ssh $e $type' output
gedge May 16, 2022
c2c4ffa
bump go version to 1.18; minor log change
gedge May 20, 2022
54170f0
config allows ~/... or ${HOME}/...; update docs/configs
gedge May 23, 2022
ea0ab2d
fix failure to update config with path using ~ or ${HOME}
gedge May 25, 2022
1a2f9cc
add 'live' tag for live/public env (prod*); update README
gedge Jun 27, 2022
4a113d0
Merge branch 'main' into feature/dp-ssh-concourse-tags
gedge Jul 18, 2022
52e91b9
bump libs
gedge Oct 13, 2022
cc85c83
Merge branch 'main' into feature/dp-ssh-concourse-tags
gedge Oct 13, 2022
ccc78ef
fix for CI in AWSB
gedge Jan 10, 2023
e835af2
bump deps
gedge Jan 10, 2023
04b3891
Merge remote-tracking branch 'origin/main' into feature/dp-ssh-concou…
gedge Jan 16, 2023
01ebdd3
update docs; switch default to aws-b
gedge Jan 22, 2023
17901f7
Merge remote-tracking branch 'origin/main' into feature/dp-ssh-concou…
gedge Feb 4, 2023
5b17d92
add user-name (cf ssh-user)
gedge Feb 4, 2023
a01239b
add nisra path,envs,repo; bump libs
gedge Feb 12, 2023
464c7f7
remove unused file .DS_Store
gedge Feb 12, 2023
4fbdcbb
rename ni-dev to nisra-dev (ditto for ni-prod)
gedge Feb 13, 2023
2a8aedc
update docs, config, libs, ioutil, sshUser
gedge Feb 14, 2023
3088c1e
Merge branch 'main' into feature/dp-ssh-concourse-tags
gedge Feb 27, 2023
f089399
bump libs
gedge Feb 27, 2023
579b92f
add audit, gofmt
gedge Feb 27, 2023
9d303d7
default env profile to its name where that env has no specific profile
gedge Mar 1, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file removed .DS_Store
Binary file not shown.
127 changes: 56 additions & 71 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ Command-line client providing *handy helper tools* for the ONS Digital Publishin

## Getting started

Clone the code (not needed if [installing on macOS](#brew-installation))
Clone the code (not needed if you [brew install on macOS](#brew-installation) :warning:)

```sh
```shell
git clone git@github.com:ONSdigital/dp-cli.git
```

Expand All @@ -20,78 +20,61 @@ git clone git@github.com:ONSdigital/dp-cli.git

The DP CLI uses Go Modules so requires a go version of **1.11** or later (ideally the latest)

Check you have a suitable version of `go` installed with:

`go version`

[ The following will ensure version 1.19

```sh
brew install go@1.19
brew unlink go
brew link --force go@1.19
```

Check desired version of `go` is on your PATH with `echo $PATH` and if not, either edit your .zshrc file to have the correct path OR do:

```sh
echo 'export GOPATH=$HOME/go' >> ~/.zshrc
echo 'export PATH="/usr/local/opt/go@1.19/bin:$PATH"' >> ~/.zshrc
```

and restart the terminal ]

Ensure `session-manager-plugin` is installed by running the following command
```

```shell
which session-manager-plugin
```
if not installed, follow this [doc](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html#install-plugin-macos)

if not installed, follow this [doc](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html#install-plugin-macos)

**Optional:**

The following are only required for some functionality of this tool.
The following are only required for some functionality of this tool.

In order to use the `dp ssh` sub-command you will need:

- [`dp-setup`](https://github.com/ONSdigital/dp-setup) cloned locally:

```bash
```shell
git clone git@github.com:ONSdigital/dp-setup
```

- [`dp-ci`](https://github.com/ONSdigital/dp-ci) cloned locally:

```bash
git clone git@github.com:ONSdigital/dp-ci
```
Note: Make sure `dp-setup` and `dp-ci` branch points to main locally. This is necessary as it has the required SSH configuration and the relavant inventories.

In order to use the `dp import cmd` sub-command (e.g. when you are using Neo4j; `import` is currently *not needed* if you are using Neptune) you will need:
Note: Make sure `dp-setup` and `dp-ci` are on the branch `main` (or `awsb` for `dp-setup`), locally. This is necessary as they have the required SSH configuration and the relevant inventories.

In order to use the `dp import cmd` sub-command (e.g. when you are using **Neo4j**; however, `import` is currently *not needed* if you are using Neptune) you will need:

- [`dp-code-list-scripts`](https://github.com/ONSdigital/dp-code-list-scripts) cloned locally:

```bash
```shell
git clone git@github.com:ONSdigital/dp-code-list-scripts
```

- [`dp-hierarchy-builder`](https://github.com/ONSdigital/dp-hierarchy-builder) cloned locally:

```bash
```shell
git clone git@github.com:ONSdigital/dp-hierarchy-builder
```

### Configuration

Configuration is defined in a YAML file. By default the CLI expects the config to be in `~/.dp-cli-config.yml`. The config file location can be customised by setting `DP_CLI_CONFIG` environment variable to your chosen path.
Configuration is defined in a YAML file. By default the CLI expects the config file to be `~/.dp-cli-config.yml`.
The config file location can be customised by setting the `DP_CLI_CONFIG` environment variable to your chosen path.

The [sample config file](./config/example_config.yml) should be tailored to suit you. For example:

```bash
```shell
cp -i config/example_config.yml ~/.dp-cli-config.yml
vi ~/.dp-cli-config.yml
```

update the paths and ssh-user:
update the paths and `ssh-user`:

```yaml
dp-setup-path: path to your local dp-setup
Expand All @@ -106,53 +89,53 @@ and if this is a first time setup, comment out `prod` from environments, thus:
```yaml
#- name: prod
# profile: dp-prod
# user: ubuntu
# tag: awsb
# user: ubuntu
```

*Note*: **ssh-user** is a string used to put your name against SecurityGroup changes.
You can uncomment them when you get `prod` access.

*Note*: **ssh-user** is a string mainly now used to put your name against SecurityGroup changes.

### Brew Installation

If using macOS, you can now install using `brew`:
If using macOS, you can install using `brew`:

- Create tap

```sh
brew tap ONSdigital/homebrew-dp-cli git@github.com:ONSdigital/homebrew-dp-cli
```
```shell
brew tap ONSdigital/homebrew-dp-cli git@github.com:ONSdigital/homebrew-dp-cli
```

- Run brew install

```sh
```shell
brew install dp-cli
```

### Build and run

Build, install and start the CLI:
If not using the *brew* installation (above), you can build, install and start the CLI thus:

```sh
```shell
make install
dp
```

[
If you get:
- If you get:

`zsh: command not found: dp`
`command not found: dp`

Then either edit your .zshrc file have the correct path OR do:
Then either edit your `~/.zshrc` file to have the correct path *or* do:

```sh
echo 'export PATH="$GOPATH/bin:$PATH"' >> ~/.zshrc
```
```shell
echo 'export PATH="$GOPATH/bin:$PATH"' >> ~/.zshrc
```

and restart the terminal ]
and restart the terminal

Or to build a binary locally:
Or to build a binary in this directory:

```sh
```shell
make build
./dp
```
Expand Down Expand Up @@ -193,35 +176,34 @@ Use the available commands for more info on the functionality available.

1. `SSOProviderInvalidToken: the SSO session has expired or is invalid`

If you see the above error, you need to re-authenticate with sign in information
If you see the above error, you need to re-authenticate with sign-in information

1. `error fetching ec2: {Name:sandbox Profile:dp-sandbox User:ubuntu Tag:awsb CI:false ExtraPorts:{Bastion:[] Publishing:[] Web:[]}}: MissingRegion: could not find region configuration`
1. `error fetching ec2: {Name:sandbox Profile:dp-sandbox SSHUser:ubuntu Tag: CI:false ExtraPorts:{Bastion:[] Publishing:[] Web:[]}}: MissingRegion: could not find region configuration`

check that you have the correct AWS profile names in your `~/.aws/config` file (dp-sandbox, dp-staging, dp-prod, dp-ci). A sample config for `~/.aws/config` is included at the end of this guide as a reference.
check that you have the correct AWS profile names in your `~/.aws/config` file (`dp-sandbox`, `dp-staging`, `dp-prod`, `dp-ci`).
A sample config for `~/.aws/config` is included at the end of this guide as a reference.

1. `Error: no security groups matching environment: "sandbox" with name "sandbox - bastion"`

check ~/.aws/credentials and remove any profile information added for dp-sandbox, dp-staging and dp-prod as this is not needed
check `~/.aws/credentials` and remove any profile information added for `dp-sandbox`, `dp-staging` and `dp-prod` as this is not needed

If you do not want to set up separate profiles, another option is to not specify any profiles in your `~/.dp-cli-config.yml`. That way the default credentials will be used.

```yaml
environments:
- name: prod
profile:
user: ubuntu
tag: awsb
user: ubuntu
- name: staging
profile:
user: ubuntu
tag: awsb
user: ubuntu
```

#### SSH/SCP command fails

```sh
$ dp ssh develop
ssh to develop
```shell
$ dp ssh sandbox
ssh to sandbox
```

If the SSH or SCP command fails, ensure that the `dp remote allow` command has been run for the environment you want to connect to.
Expand All @@ -234,6 +216,7 @@ Ensure you have `region=eu-west-2` in your AWS configuration.

Depending on the command you're trying to run, and what you're trying to access, ensure your `AWS_PROFILE` is set correctly and there is no prod/sandbox/ci config added in the `~/.aws/credentials` file.
Example:

```yaml
export AWS_PROFILE=dp-staging
```
Expand Down Expand Up @@ -269,7 +252,7 @@ $ dp ssh sandbox web 1 date
:warning: However, if you wish to include *flags* in the (remote) command, you must tell `dp` to stop looking for flags - use the `--` flag:

```sh
$ dp ssh develop web 1 -- ls -la
$ dp ssh sandbox web 1 -- ls -la
[...]
```

Expand All @@ -287,8 +270,8 @@ MY_IP=192.168.11.22 dp remote allow sandbox

Similarly, use the `--user` flag to change the label attached to the IP that is put into (or removed from) the *allow* table.

```sh
dp remote --user MyColleaguesName --ip 192.168.44.55 --http-only allow develop
```shell
dp remote --user MyColleaguesName --ip 192.168.44.55 --http-only allow sandbox
```

#### Remote allow extra ports
Expand All @@ -302,23 +285,25 @@ environments:
publishing:
- 80
```

#### AWS Command Line Access

Follow the guide in [dp](https://github.com/ONSdigital/dp/blob/main/guides/AWS_ACCOUNT_ACCESS.md)

## Releases

When creating new releases, please be sure to:

- update the version (tag)
- update the brew formula [in the tap](https://github.com/ONSdigital/homebrew-dp-cli).

## Sample config for `~/.aws/config`:
## Sample config for `~/.aws/config`

```
```ini
[profile dp-sandbox]
sso_start_url = https://ons.awsapps.com/start
sso_account_id = 1234556253 #replace this with correct account id
sso_role_name = AdministratorAccess
sso_region = eu-west-2
region = eu-west-2
```
```
Loading