New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support pick-and-choose modular services #746

Closed
cpu opened this Issue Jun 20, 2017 · 19 comments

Comments

Projects
None yet
2 participants
@cpu
Member

cpu commented Jun 20, 2017

This issue is an "omnibus" issue to track first party support for pick-and-choose ("modular") Streisand services. This issue will track the proposed design & the overall progress made to convert individual services/roles.

Motivation

Historically Streisand's anti-censorship goal has been the driving force behind providing many services pre-configured out-of-box. For users on censored & hostile networks it can be difficult to know what will work ahead of time and often the "best" or "most secure" choices are flat out unusable. This userbase continues to need a service like Streisand provides.

On the other hand, the recent Internet climate w.r.t Internet Service Provider surveillance has driven increased demand for a simple do-it-yourself VPN service usable on traditional uncensored networks, leading many to evaluate Streisand. From the perspective of this use-case Streisand's attack surface & complexity are a disadvantage and a risk.

How can a balance be struck? By allowing services to be enabled/disabled, both use-cases can be addressed. For the user's already served by Streisand who need to balance what works with the "academically pure" solution, all services can be enabled/included. For users with a more narrow & straightforward use-case just a bare minimum of the services they require can be included.

Challenge

The difficulty of providing this modularity lies in the interconnections between roles that exists today. Simply disabling the Shadowsocks role (for example) cascades out into other roles that have implicit dependencies on variables & files that the Shadowsocks role populates & creates. The documentation makes reference to Shadowsocks variables. The mirror role assumes that Shadowsocks clients are required. The firewall role requires knowledge of Shadowsocks to enable/disable port mappings.

The most immediate & straight forward way to address this challenge is to pepper the various roles with if statements evaluating whether or not a role was included. This approach (or variations of it) have been proposed in community contributed PRs in the past, but the complexity and overall brittleness of this approach make it hard to manage.

For this Issue we intend to exploit the relative similarity of the needs of a Streisand service to encourage isolated role design. Put simply, each service's role will be required to do its own firewall updates, its own documentation generation, and its own client mirroring. Base firewall/documentation roles will fill in the gaps by providing required system-wide dependencies (the ufw package being installed for instance) and stitch together index documentation based on which services were enabled/disabled.

Base System

I consider the following to be the "base system" (e.g. these are not things you can pick-and-choose). These pieces are either integral to the goal of Streisand (things like a documentation web portal) or required for generic system access/administration (e.g. ufw, ssh).

  • ufw
  • nginx
  • monit (likely to be replaced with better systemd units)
  • ssh (the base sshd & config)
  • the gateway docs
  • the client mirror (this might be something that would make sense to allow toggling for)

Modular Services

Services/roles to modularize/update:

  • - shadowsocks - merged #794
  • - tor - merged #808
  • - openvpn - merged #816
  • - l2tp-ipsec - merged #896
  • - openconnect - merged #834
  • - ssh fowarding user - merged #871
  • - tinyproxy - merged #845
  • - sslh? - going to punt on this one for now and revisit later.
  • - wireguard - merged #802

Additional tasks:

  • Document the zen/approach we're adopting for writing Streisand modules
  • Integrate module/service selection into the streisand wrapper script
  • Output some kind of "manifest" of user choices that can be referenced in support issues - covered by #926
  • Implement a testing strategy. Ideally we'd test every possible enabled/disabled configuration but that's ~n^2 - merged #988
  • Update English documentation to describe modularity/service choices (& maybe suggest useful "profiles" of choices?) - punted to #1041

@cpu cpu self-assigned this Jun 20, 2017

@cpu

This comment has been minimized.

Member

cpu commented Jun 20, 2017

I've started some proof-of-concept work modularizing Shadowsocks in a branch: cpu@039ccc1

This branch also has an initial attempt at providing documentation for the modularization design: https://github.com/cpu/streisand/blob/cpu-optional-shadowsocks/documentation/modular_roles.md

There are outstanding questions to answer about things like dnsmasq and other shared infrastructure that bridges multiple roles. I'm hoping to get into the weeds with these things in the coming week(s).

@cpu

This comment has been minimized.

Member

cpu commented Jun 25, 2017

Finished up with making WireGuard optional in a branch cut off of the cpu-optional-shadowsocks branch mentioned above: https://github.com/cpu/streisand/tree/cpu-optional-wireguard

Doing this cleanly required a fairly major retuning of the dnsmasq role: cpu@bab644b The approach used is documented here.

Similarly, the rc-local role was removed entirely and replaced by individual services managing their own /etc/init.d/ scripts describing their own runlevels & inter-dependency needs: cpu@56b53ab

@cpu

This comment has been minimized.

Member

cpu commented Jul 12, 2017

The first modular role (shadowsocks) was merged to master in #794.
I'm going to move on to cleaning up the wireguard branch for master next.

cpu added a commit that referenced this issue Jul 16, 2017

Modular Streisand: Optional Wireguard (#802)
This commit updates the Wireguard role to be optional, allowing it to be enabled/disabled by changing the `playbooks/group_vars/all` variable `streisand_wireguard_enabled` var. By default it is true and the base Streisand system remains unchanged.

Compared to #794 this was a more involved conversion so I've included some notes on required changes this commit adds along the way:

#### Support modular dnsmasq config.

Prior to this commit the `dnsmasq` role had an implicit dependency on both the `openvpn` role and the `wireguard` role. Listen address variables were referenced from the centralized `/etc/dnsmasq.conf` file in a single comma separated `listen-address` config parameter.

This commit changes the `/etc/dnsmasq.conf` template to only specify `listen-address=127.0.0.1` for the localhost bind. Subsequently the `openvpn` and `wireguard` roles are updated to include tasks that write
their own individual `openvpn.conf` and `wireguard.conf` files to `/etc/dnsmasq.d/` where they will be read in and amalgamated into one config at runtime. This allows roles to augment the base `dnsmasq` listeners when they need to without the `dnsmasq` role having to know about it.

#### Remove rc-local role in favour of modular approach.

Similarly, to avoid the one `rc-local` role having cross dependencies this commit removes it entirely. Instead individual roles can write their own service description to /etc/init.d/ and enable it using the standard mechanisms.

The `ip-forwarding`, `l2tp-ipsec`, `openconnect`, `openvpn` and `wireguard` roles are updated accordingly.

#### Documentation 

This commit updates `documentation/modular_roles.md` to describe the approach taken above. 

Updates #746
@cpu

This comment has been minimized.

Member

cpu commented Jul 16, 2017

Second modular role (wireguard) was merged to master in #802.

@cpu

This comment has been minimized.

Member

cpu commented Jul 16, 2017

Submitted a PR for the Tor role: #808

@alimakki alimakki referenced this issue Jul 19, 2017

Merged

Modular Streisand: Optional OpenVPN and stunnel #816

4 of 6 tasks complete
@alimakki

This comment has been minimized.

Collaborator

alimakki commented Jul 19, 2017

Submitted a PR for the OpenVPN role: #816

cpu added a commit that referenced this issue Jul 23, 2017

Modular Streisand: Optional Tor (#808)
This commit updates the Tor role to be optional, allowing it to be enabled/disabled by changing the `playbooks/group_vars/all` variable `streisand_tor_enabled`. By default it is true and the base Streisand system remains unchanged.

This largely follows in the footsteps of #794 and #802 and doesn't have much of note. The only "tricky" bit was removing the Nginx hidden service vhost from the `streisand-gateway` role. It is now a separate vhost templated into `/etc/nginx/sites-available` (and `sites-enabled`) under the name `streisand-hidden-service`. Now it is only created & enabled when the Tor role is applied.

Updates #746
@cpu

This comment has been minimized.

Member

cpu commented Jul 23, 2017

Modular Tor role merged to master 🎉

@cpu

This comment has been minimized.

Member

cpu commented Jul 25, 2017

Modular OpenVPN role merged to master thanks to @alimakki 🌮 🥇

@alimakki alimakki referenced this issue Jul 26, 2017

Merged

Modular Streisand: Optional ocserv #834

4 of 6 tasks complete
@alimakki

This comment has been minimized.

Collaborator

alimakki commented Jul 26, 2017

Submitted a PR for the OpenConnect role: #834

cpu added a commit that referenced this issue Jul 30, 2017

Optional ocserv (#834)
This PR modifies Streisand to make the OpenConnect role optional, which can be configured by modifying the `streisand_openconnect_enabled` to `no`, located in `global_vars/vars.yml`.

Updates #746.
@cpu

This comment has been minimized.

Member

cpu commented Jul 30, 2017

Modular OpenConnect merged to master. Thanks @alimakki 🏆 🍹

This was referenced Jul 30, 2017

@alimakki

This comment has been minimized.

Collaborator

alimakki commented Aug 2, 2017

Modular tinproxy merged!

@cpu cpu referenced this issue Aug 5, 2017

Merged

Modular Streisand: Optional SSH Forward User #871

3 of 6 tasks complete
@cpu

This comment has been minimized.

Member

cpu commented Aug 5, 2017

Opened a PR for a modular SSH forward user #871

@cpu

This comment has been minimized.

Member

cpu commented Aug 9, 2017

Modular SSH forward user merged! Another one down 🏁

@cpu cpu referenced this issue Aug 20, 2017

Merged

Modular Streisand: optional l2tp/ipsec. #896

4 of 6 tasks complete
@cpu

This comment has been minimized.

Member

cpu commented Aug 20, 2017

Opened a PR for l2tp: #896

cpu added a commit that referenced this issue Aug 28, 2017

Modular Streisand: optional l2tp/ipsec. (#896)
This commit introduces modular support for l2tp/ipsec. The role is
included/skipped based on the var `streisand_l2tp_enabled`, which by
default is 'true'.

The `rsyslog` role is now removed since both `openconnect` and `l2tp`
modify the configuration as required individually. I moved the restart
handler into each role to avoid the need for a centralized `rsyslog`
role.

Updates #746
@cpu

This comment has been minimized.

Member

cpu commented Aug 28, 2017

Modular l2tp/ipsec merged. I think that's all of the service roles completed!

@cpu

This comment has been minimized.

Member

cpu commented Sep 7, 2017

Opened a PR to track a manifest of which roles are selected as well as other useful diagnostic information; #926

cpu added a commit that referenced this issue Sep 22, 2017

Allow customizing Streisand services at install time. (#936)
This commit updates Streisand to allow enabling/disabling the core
services at install time, or by customizing a site specific config file
before running Streisand.

Ansible's prompt functionality is unfortunately crippled with respect to
templating and `when` conditions. This makes it tricky to accomplish the
customization we want where you are only prompted for each service if
you want to customize the overall installation.

To allowing changing services on a per-site basis & remembering the
decisions the wrapper script now creates a `$HOME/.streisand` directory
with a `$HOME/.streisand/site.yml` site specific config file. By
default it is populated with the `global_vars/default-site.yml` file
from the Streisand repository.

The `streisand` wrapper conditionally invokes a separate playbook
(`playbooks/customize.yml`) for customization that rewrites the
`$HOME/.streisand/site.yml` vars file based on what the user specifies.
It's a little bit ugly but It Works(!). Further refinement welcome!

This methodology also supports customizing the installed services
non-interactively by editing `$HOME/.streisand/site.yml` ahead of
running `./streisand` and skipping the customization step. This is
useful if (for example) you only ever want to install Wireguard on your
Streisand instances. You can create a `$HOME/.streisand/site.yml` that
only enables Wireguard and all of your Streisand instances will be
provisioned accordingly.

The validation role is run after customization to ensure that the
choices are valid and don't result in (for e.g.) no services enabled.

Presently both Travis and the Vagrant `streisand-host` ignore the
`$HOME/.streisand.yml` and use the `global_vars/default-site.yml` vars
resulting in a provision with all services enabled.

Updates #746
@cpu

This comment has been minimized.

Member

cpu commented Sep 22, 2017

Almost there folks! I just merged #936 and we hit an important milestone. You can now interactively enable/disable Streisand services when you run ./streisand. 🎉

The only things that remain are better testing of different enabled/disabled combinations & a README/docs update.

@cpu

This comment has been minimized.

Member

cpu commented Oct 14, 2017

I've implemented a testing strategy for different configurations of Streisand services in #988

cpu added a commit that referenced this issue Oct 16, 2017

Modular Streisand: improved test coverage of enabled/disabled services (
#988)

This PR updates the `tests/tests.sh` driver script so that different build configurations can be run as subtasks depending on a provided env var. The `.travis.yml` configuration is updated to use a matrix of env vars such that we do several subtasks for every CI run. Presently there are 6 build subtasks:

1. A build with everything enabled & all of the syntax checking (the same as was done prior to this commit for all CI runs)
2. A build with just l2tp enabled
3. A build with just openvpn enabled
4. A build with just shadowsocks enabled
5. A build with just the SSH forward user enabled
6. A build with the enabled/disabled services randomized

Care was taken with the randomized build to ensure that there is always 1 service enabled and that none of the dependent services end up in a state that the validation role will reject (e.g. stunnel without openvpn). It is my hope that with the random build we will get some good coverage over time, especially with the daily cron build.

The `development-setup.yml` playbook was updated to only install/build Libreswan on the container host if the container will use l2tp. This should help speed up non-l2tp builds.

Updates #746
@cpu

This comment has been minimized.

Member

cpu commented Nov 5, 2017

I've been dragging my feet on a README update because of the translation impact. Since the rest of the work is completed I cut a separate issue for the README update: #1041

We can officially call this closed! You can now pick and choose which services will be configured for your Streisand host. Thanks to @alimakki for all the help! This is a real milestone :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment