-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Uses local RPM build for "dev" and "staging" scenarios #587
Changes from 11 commits
500cddb
abd7e98
e85be7e
69f995c
2a31b21
275b60c
715462c
dfca6d0
58cedea
9dc287a
902fc18
a7f57d1
bce65c8
087c709
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -116,7 +116,9 @@ Select all VMs marked as **updates available**, then click **Next**. Once all up | |
|
||
#### Download, Configure, Copy to `dom0` | ||
|
||
Decide on a VM to use for development. We suggest creating a standalone VM called `sd-dev`. Clone this repo to your preferred location on that VM. | ||
Decide on a VM to use for development. We recommend creating a standalone VM called `sd-dev` by following [these instructions](https://docs.securedrop.org/en/stable/development/setup_development.html#qubes). You must install Docker in that VM in order to build a development environment using the standard workflow. | ||
|
||
Clone this repo to your preferred location on that VM. | ||
|
||
Next we need to do some SecureDrop-specific configuration: | ||
|
||
|
@@ -139,7 +141,7 @@ After that initial manual step, the code in your development VM may be copied in | |
[dom0]$ export SECUREDROP_DEV_VM=sd-dev # set to your dev VM | ||
[dom0]$ export SECUREDROP_DEV_DIR=/home/user/projects/securedrop-workstation # set to your working directory | ||
[dom0]$ cd ~/securedrop-workstation/ | ||
[dom0]$ make clone # copy repo to dom0 | ||
[dom0]$ make clone # build RPM package (requires Docker) and copy repo to dom0 | ||
``` | ||
|
||
If you plan to work on the [SecureDrop Client](https://github.com/freedomofpress/securedrop-client) code, also run this command in `dom0`: | ||
|
@@ -178,8 +180,6 @@ When the installation process completes, a number of new VMs will be available o | |
|
||
The staging environment is intended to provide an experience closer to a production environment. For example, it will alter power management settings on your laptop to prevent suspending it to disk, and make other changes that may not be desired during day-to-day development in Qubes. | ||
|
||
**IMPORTANT: THE STAGING ENVIRONMENT SHOULD NEVER BE USED FOR PRODUCTION PURPOSES. IT SHOULD ALSO NOT BE USED ON DEVELOPER MACHINES, BUT ONLY ON TEST MACHINES THAT HOLD NO SENSITIVE DATA.** | ||
|
||
#### Update `dom0`, `fedora-31`, `whonix-gw-15` and `whonix-ws-15` templates | ||
|
||
Updates to these VMs will be provided by the installer and updater, but to ensure they are up to date prior to install, it will be easier to debug, should something go wrong. | ||
|
@@ -192,9 +192,9 @@ In the Qubes Menu, navigate to `System Tools` and click on `Qubes Update`. Click | |
|
||
You can install the staging environment in two ways: | ||
|
||
- If you have an up-to-date clone of this repo with a valid configuration in `dom0`, you can use the `make staging` target to provision a staging environment. Prior to provisioning, `make staging` will set your `config.json` environment to `staging`. As part of the provisioning, your package repository configuration will be updated to use the latest test release of the RPM package, and the latest nightlies of the Debian packages. | ||
- If you have an up-to-date clone of this repo with a valid configuration in `dom0`, you can use the `make staging` target to provision a staging environment. Prior to provisioning, `make staging` will set your `config.json` environment to `staging`. As part of the provisioning, a locally built RPM will be installed in dom0, and your package repository configuration will be updated to use the latest test release of the RPM package, and the latest nightlies of the Debian packages (same as `make dev`). | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
This could be somewhat misleading. If the version of the locally built package is equal or higher than the one mirrored by yum-test, the latest test release of the RPM package will not be used (but it will receive test package updates). We should specify to use the prod install procedures to fully mirror the prod flow (or ensure the RPM is build off the latest tag/rc when cloning the repo in the dev VM). (related to comment in https://github.com/freedomofpress/securedrop-workstation/pull/587/files#r458880338) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Fair point. Rephrased as:
|
||
|
||
- If you want to install a staging environment from scratch in a manner similar to a production install (starting from an RPM, and using `sdw-admin` for the installation), follow the process in the following sections. | ||
- If you want to download a specific version of the RPM, and follow a verification procedure similar to that used in a production install, follow the process in the following sections. | ||
|
||
#### Download and install securedrop-workstation-dom0-config package | ||
|
||
|
@@ -274,7 +274,7 @@ In a terminal in `dom0`, run the following commands: | |
|
||
This project's development requires different workflows for working on provisioning components and working on submission-handling scripts. | ||
|
||
For developing salt states and other provisioning components, work is done in a development VM and changes are made to individual state and top files there. In the `dom0` copy of this project, `make clone` is used to copy over the updated files; `make <vm-name>` to rebuild an individual VM; and `make dev` to rebuild the full installation. Current valid target VM names are `sd-proxy`, `sd-gpg`, `sd-whonix`, and `disp-vm`. Note that `make clone` requires two environment variables to be set: `SECUREDROP_DEV_VM` must be set to the name of the VM where you've been working on the code, the `SECUREDROP_DEV_DIR` should be set to the directory where the code is checked out on your development VM. | ||
For developing salt states and other provisioning components, work is done in a development VM and changes are made to individual state and top files there. In the `dom0` copy of this project, `make clone` is used to package and copy over the updated files; `make <vm-name>` to rebuild an individual VM; and `make dev` to rebuild the full installation. Current valid target VM names are `sd-proxy`, `sd-gpg`, `sd-whonix`, and `disp-vm`. Note that `make clone` requires two environment variables to be set: `SECUREDROP_DEV_VM` must be set to the name of the VM where you've been working on the code, the `SECUREDROP_DEV_DIR` should be set to the directory where the code is checked out on your development VM. | ||
|
||
For developing submission processing scripts, work is done directly in the virtual machine running the component. To commit, copy the updated files to a development VM with `qvm-copy-to-vm`and move the copied files into place in the repo. (This process is a little awkward, and it would be nice to make it better.) | ||
|
||
|
@@ -298,7 +298,7 @@ Be aware that running tests *will* power down running SecureDrop VMs, and may re | |
|
||
Double-clicking the "SecureDrop" desktop icon will launch a preflight updater that applies any necessary updates to VMs, and may prompt a reboot. | ||
|
||
To update workstation provisioning logic, one must use the `sd-dev` AppVM that was created during the install. From your checkout directory, run the following commands (replace `<tag>` with the tag of the release you are working with): | ||
To update workstation provisioning logic in a development environment, one must use the `sd-dev` AppVM that was created during the install. From your checkout directory, run the following commands (replace `<tag>` with the tag of the release you are working with): | ||
|
||
``` | ||
git fetch --tags | ||
|
@@ -313,7 +313,7 @@ make clone | |
make dev | ||
``` | ||
|
||
In the future, we plan on shipping a *SecureDrop Workstation* installer package as an RPM package in `dom0` to automatically update the salt provisioning logic. | ||
The `make clone` command will build a new version of the RPM package that contains the provisioning logic in your development VM (e.g., `sd-dev`) and copy it to `dom0`. The RPM is built using a Docker container, so Docker must be installed in your development VM. | ||
|
||
### Building the Templates | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -198,21 +198,13 @@ dom0-securedrop-launcher-desktop-shortcut: | |
- mode: 755 | ||
|
||
{% import_json "sd/config.json" as d %} | ||
{% if d.environment == "dev" %} | ||
emkll marked this conversation as resolved.
Show resolved
Hide resolved
|
||
dom0-remove-securedrop-workstation-dom0-config: | ||
pkg.removed: | ||
- pkgs: | ||
- securedrop-workstation-dom0-config | ||
|
||
{% else %} | ||
|
||
{% if d.environment != "dev" %} | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If I understand correctly, here we are explicitly excluding dev from installing the RPM to avoid installing the latest from the yum repos should the locally built version be lesser than the ones on the server. if that's the case, it may be worth adding a comment here for future maintainers as it is somewhat counter-intuitive. |
||
dom0-install-securedrop-workstation-dom0-config: | ||
pkg.installed: | ||
- pkgs: | ||
- securedrop-workstation-dom0-config | ||
- require: | ||
- file: dom0-workstation-rpm-repo | ||
|
||
{% endif %} | ||
|
||
# Hide suspend/hibernate options in menus in prod systems | ||
|
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,12 +20,19 @@ dev_dir="${SECUREDROP_DEV_DIR:-/home/user/securedrop-workstation}" | |
# The dest directory in dom0 is not customizable. | ||
dom0_dev_dir="$HOME/securedrop-workstation" | ||
|
||
# Call out to target AppVM, to build an RPM containing | ||
# the latest Salt config for dom0. The RPM will be included | ||
# in the subsequent tarball, which is fetched to dom0. | ||
function build-dom0-rpm() { | ||
printf "Building RPM on %s ...\n" "${dev_vm}" | ||
qvm-run -q "$dev_vm" "make -C $dev_dir dom0-rpm" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. would it make sense here to bump the RPM version to make sure that the version is always higher than anything available? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not aware of any RPM tooling like |
||
} | ||
|
||
# Call out to target AppVM to create a tarball in dom0 | ||
function create-tarball() { | ||
printf "Cloning code from %s:%s ...\n" "${dev_vm}" "${dev_dir}" | ||
qvm-run --pass-io "$dev_vm" \ | ||
"tar -c --exclude-vcs \ | ||
--exclude='*.rpm' \ | ||
-C '$(dirname "$dev_dir")' \ | ||
'$(basename "$dev_dir")'" > /tmp/sd-proj.tar | ||
} | ||
|
@@ -35,5 +42,6 @@ function unpack-tarball() { | |
tar xf /tmp/sd-proj.tar -C "${dom0_dev_dir}" --strip-components=1 | ||
} | ||
|
||
build-dom0-rpm | ||
create-tarball | ||
unpack-tarball |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The changes to the dev environment may have repercussions on these individual makefile targets :
prep-salt
makefile target would copy the local salt file in/srv/salt/
whereas this updatedprep-dev
target will install the dom0 rpm. This means that any changes to the local files in thesecuredrop-workstation
folder in dom0 will not be used.If this is the case, adding a note to this effect in the dev docs could be helpful
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's definitely true. Editing the files in e.g. /srv/salt/ still works fine, but I wouldn't recommend either, given how easy it is to lose changes that way. Will clarify in the docs!