Skip to content

Commit

Permalink
docs: update buildkite installation to include icinga (#5802) [skip ci]
Browse files Browse the repository at this point in the history 
Co-authored-by: Stanislav Zhuk <stasadev@gmail.com>
  • Loading branch information
@rfay @stasadev
rfay and stasadev committed Feb 11, 2024
1 parent 42e61d3 commit 8c7af56
Show file tree
Hide file tree
Showing 2 changed files with 46 additions and 42 deletions.
4 changes: 2 additions & 2 deletions .buildkite/testbot_maintenance.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ fi
# Upgrade various items on various operating systems
case $os in
darwin)
for item in ddev/ddev-edge/ddev golang golangci-lint libpq mkcert mkdocs; do
for item in ddev/ddev/ddev golang golangci-lint libpq mkcert mkdocs; do
brew upgrade $item || brew install $item || true
done
brew link --force libpq
Expand All @@ -39,7 +39,7 @@ windows)
linux)
# homebrew is only on amd64
if [ "$(arch)" = "x86_64" ]; then
for item in ddev/ddev-edge/ddev golang libpq mkcert mkdocs; do
for item in golang libpq mkcert mkdocs; do
brew upgrade $item || brew install $item || true
done
brew link --force libpq
Expand Down
84 changes: 44 additions & 40 deletions docs/content/developers/buildkite-testmachine-setup.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@ We are using [Buildkite](https://buildkite.com/ddev) for Windows and macOS testi

```bash
@echo off
set DOCKERHUB_PULL_USERNAME=druddockerpullaccount
set DOCKERHUB_PULL_PASSWORD=xxx_readonly_token
```

Expand All @@ -39,23 +38,27 @@ We are using [Buildkite](https://buildkite.com/ddev) for Windows and macOS testi

## Additional Windows Setup for WSL2+Docker Desktop Testing

1. Do not set up `buildkite-agent` on the Windows side, or disable it.
2. Edit Ubuntu's `/etc/wsl.conf` to contain:
1. The Ubuntu distro should be set up with the user `buildkite-agent`
2. Do not set up `buildkite-agent` on the Windows side, or disable it.
3. Edit Ubuntu's `/etc/wsl.conf` to contain:

```
[boot]
systemd=true
```

3. Update WSL2 to WSL2 Preview from Microsoft Store and `wsl --shutdown` and then restart.
4. `wsl --update`
5. Open WSL2 and check out [ddev/ddev](https://github.com/ddev/ddev).
6. As normal user, run `.github/workflows/linux-setup.sh`.
7. `export PATH=/home/linuxbrew/.linuxbrew/bin:$PATH
echo "export PATH=/home/linuxbrew/.linuxbrew/bin:$PATH" >>~/.bashrc`
7. Configure brew in PATH with:

```
echo "export PATH=/home/linuxbrew/.linuxbrew/bin:$PATH" >>~/.bashrc
source ~/.bashrc
```

8. As root user, add sudo capability with `echo "ALL ALL=NOPASSWD: ALL" >/etc/sudoers.d/all && chmod 440 /etc/sudoers.d/all`.
9. Manually run `testbot_maintenance.sh`, `curl -sL -O https://raw.githubusercontent.com/ddev/ddev/master/.buildkite/testbot_maintenance.sh && bash testbot_maintenance.sh`.
9. Manually run `testbot_maintenance.sh`, `.buildkite/testbot_maintenance.sh`.
10. `git config --global --add safe.directory '*'`
11. Install basics in WSL2:

Expand All @@ -67,34 +70,26 @@ We are using [Buildkite](https://buildkite.com/ddev) for Windows and macOS testi

sudo mkdir -p /usr/sharekeyrings && curl -fsSL https://keys.openpgp.org/vks/v1/by-fingerprint/32A37959C2FA5C3C99EFBC32A79206696452D198 | sudo gpg --dearmor -o /usr/share/keyrings/buildkite-agent-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/buildkite-agent-archive-keyring.gpg] https://apt.buildkite.com/buildkite-agent stable main" | sudo tee /etc/apt/sources.list.d/buildkite-agent.list
sudo apt update && sudo apt install -y build-essential buildkite-agent ca-certificates curl ddev gnupg lsb-release make mariadb-client
sudo apt update && sudo apt install -y build-essential buildkite-agent ca-certificates curl ddev etckeeper gnupg icinga2 nagios-plugins lsb-release make mariadb-client
sudo snap install ngrok
```

12. [Configure `buildkite-agent` in WSL2](https://buildkite.com/docs/agent/v3/ubuntu). It needs the same changes as macOS, but tags `tags="os=wsl2,architecture=amd64,dockertype=dockerforwindows"` and build-path should be in `~/tmp/buildkite-agent`.

13. The buildkite/hooks/environment file must be updated to contain the Docker pull credentials:

```bash
#!/bin/bash
export DOCKERHUB_PULL_USERNAME=druddockerpullaccount
export DOCKERHUB_PULL_PASSWORD=xxx_readonly_token
set -e
```

14. Verify that `buildkite-agent` is running.
15. In Task Scheduler, create a task that runs on User Logon and runs `C:\Windows\System32\wsl.exe` with arguments `-d Ubuntu`.
16. Add `buildkite-agent` to the `docker` and `testbot` groups in `/etc/group`
17. `echo "capath=/etc/ssl/certs/" >>~/.curlrc` And then do the same as `buildkite-agent` user
18. `sudo chmod -R ug+w /home/linuxbrew`
19. `nc.exe -l -p 9003` on Windows to trigger and allow Windows Defender.
20. Run `ngrok config add-authtoken <token>` with token for free account.
21. Copy ngrok config into `buildkite-agent` account, `sudo cp -r ~/.ngrok2 ~buildkite-agent/ && sudo chown -R buildkite-agent:buildkite--agent ~buildkite-agent/ngrok2`
22. Add `/home/linuxbrew/.linuxbrew/bin` to `PATH` in `/etc/environment`.
23. Copy ngrok config into `buildkite-agent` account, `sudo cp -r ~/.ngrok2 ~buildkite-agent/ && sudo chown -R buildkite-agent:buildkite--agent ~buildkite-agent/ngrok2`
24. Add `buildkite-agent` to `sudo` group in `/etc/groups`
25. Give `buildkite-agent` a password with `sudo passwd buildkite-agent`
26. As `buildkite-agent` user `mkcert -install`
13. Verify that `buildkite-agent` is running.
14. Install icinga2, `sudo apt install -y icinga2`
15. Follow the [Icinga instructions](https://newmonitor.thefays.us/icingaweb2/doc/module/director/chapter/Working-with-agents) to configure the agent. Under the "Agent" tab it provides a script to configure the agent.
16. In Task Scheduler, create a task that runs on User Logon and runs `C:\Windows\System32\wsl.exe` with arguments `-d Ubuntu`.
17. Add `buildkite-agent` to the `docker` and `testbot` groups in `/etc/group`
18. `echo "capath=/etc/ssl/certs/" >>~/.curlrc` And then do the same as `buildkite-agent` user
19. `sudo chmod -R ug+w /home/linuxbrew`
20. `nc.exe -l -p 9003` on Windows to trigger and allow Windows Defender.
21. Run `ngrok config add-authtoken <token>` with token for free account.
22. Copy ngrok config into `buildkite-agent` account, `sudo cp -r ~/.ngrok2 ~buildkite-agent/ && sudo chown -R buildkite-agent:buildkite--agent ~buildkite-agent/ngrok2`
23. Add `/home/linuxbrew/.linuxbrew/bin` to `PATH` in `/etc/environment`.
24. Copy ngrok config into `buildkite-agent` account, `sudo cp -r ~/.ngrok2 ~buildkite-agent/ && sudo chown -R buildkite-agent:buildkite--agent ~buildkite-agent/ngrok2`j
25. Add `buildkite-agent` to `sudo` group in `/etc/groups`
26. Give `buildkite-agent` a password with `sudo passwd buildkite-agent`
27. As `buildkite-agent` user `mkcert -install`

## Additional Windows Setup for WSL2+Docker-Inside Testing

Expand All @@ -106,7 +101,7 @@ We are using [Buildkite](https://buildkite.com/ddev) for Windows and macOS testi
sudo mkdir -p /etc/apt/keyrings
sudo mkdir -p /etc/apt/keyrings && sudo rm -f /etc/apt/keyrings/docker.gpg && curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update && sudo apt install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo apt update && sudo apt install -y docker-ce docker-ce-cli etckeeper containerd.io docker-compose-plugin
sudo usermod -aG docker $USER
```

Expand All @@ -117,13 +112,23 @@ We are using [Buildkite](https://buildkite.com/ddev) for Windows and macOS testi

```
#!/bin/bash
export DOCKERHUB_PULL_USERNAME=druddockerpullaccount
export DOCKERHUB_PULL_PASSWORD=xxx_readonly_token
export CAROOT=/mnt/c/Users/testbot/AppData/Local/mkcert
set -e
```

5. Run `.buildkite/sanetestbot.sh`

## Icinga2 monitoring setup for WSL2 instances

1. Icinga Director web UI, configure the host on `newmonitor.thefays.us`, normally making a copy of an existing identical item.
2. Deploy the new host using Icinga Director.
3. On the WSL2 Ubuntu instance, install needed packages: `sudo apt update && sudo apt install -y etckeeper icinga2 monitoring-plugins-contrib nagios-plugins`
4. Add `nagios` to the `docker` group in `/etc/group`.
5. `sudo icinga2 node wizard` to configure the agent, see [docs](https://icinga.com/docs/icinga-2/latest/doc/06-distributed-monitoring/#agentsatellite-setup-on-linux)
6. Restart `sudo systemctl restart icinga2`
7. Hope that it can all work sometime.

## macOS Test Agent Setup (Intel and Apple Silicon)

1. Create the user “testbot” on the machine. Use the password for `ddevtestbot@gmail.com`, available in 1Password.
Expand All @@ -139,7 +144,7 @@ We are using [Buildkite](https://buildkite.com/ddev) for Windows and macOS testi
11. Install [Homebrew](https://brew.sh/) `/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"`
12. After installing Homebrew follow the instructions it gives you at the end to add brew to your PATH.
13. Install everything you’ll need with `brew install buildkite/buildkite/buildkite-agent bats-core composer ddev/ddev/ddev git golang jq mariadb mkcert netcat p7zip && brew install --cask docker iterm2 ngrok`.
14. Run `ngrok config add-authtoken <token>` with token for free account from 1Password.
14. Run `ngrok authtoken <token>` with token for free account from 1Password.
15. Run `mkcert -install`.
16. If Docker Desktop will be deployed, run Docker manually and go through its configuration routine.
17. If OrbStack will be deployed, install it from [orbstack.dev](https://orbstack.dev).
Expand All @@ -159,12 +164,11 @@ We are using [Buildkite](https://buildkite.com/ddev) for Windows and macOS testi
* `build-path="~/tmp/buildkite-agent/builds"`
24. The `buildkite-agent/hooks/environment` file must be created and set executable to contain the Docker pull credentials (found in `druddockerpullaccount` in 1Password):

```bash
#!/bin/bash
export DOCKERHUB_PULL_USERNAME=druddockerpullaccount
export DOCKERHUB_PULL_PASSWORD=xxx_readonly_token
set -e
```
```
#!/bin/bash
export DOCKERHUB_PULL_PASSWORD=xxx_readonly_token
set -e
```

25. Run `brew services start buildkite-agent`.
26. Run `bash ~/workspace/ddev/.buildkite/testbot_maintenance.sh`.
Expand Down

0 comments on commit 8c7af56

Please sign in to comment.