Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/nats fw 2.268 #271

Merged
merged 2 commits into from
Mar 29, 2022
Merged

Feature/nats fw 2.268 #271

merged 2 commits into from
Mar 29, 2022

Conversation

nouseforaname
Copy link
Contributor

@nouseforaname nouseforaname commented Mar 23, 2022

Backported commits in main branch for:

  • add updated interface to centos, rhel etc. netmanagers
  • recreate fakes
  • go mod tidy
  • go mod vendor

ran unit-tests

#259

    Add interfaces and implementations for Nats FW Win

    Add code to the agent to secure the remote mbus that
    bosh provides. Provide a mechanism to restrict workloads
    on the VM other than the agent itself to connect to BOSHs
    mbus.

    On VM creation bosh provides the Agent on the VM with creds
    to connect to Nats. Worklaoads that live on the vm could
    extract these credentials and use them to connect to nats and
    extract further credentials that are being sent over nats once
    the rendering of jobs takes place.

    This creates a FW rule within the agent bootstrap to limit which
    binaries can create outgoing requests to the nats endpoint.

    The created FW rules are ephemeral and exist as long as the agent
    process runs. This avoids the necessity for cleaning up FW rules if
    the remote address of the nats should change. If a process starts
    before the rule exists, the rule will still apply once it exists.

    Updated tests relying on SetupNetwork calls with new method signature

#264

This commit adds the SetupFirewall implemention for Linux agents.

unit tests are passing, integration tests are passing (ran them multiple
times in a row to rule out flakyness because of the complicated setup)

functionality
why return early if mbusUrl is ""?
Because on some IaaSs we require calling SetupNetwork multiple times.
The first time it is called we cannot possible have the mbus url known and
therefore needed a way to make this idempotent. Specifically [example here](https://github.com/cloudfoundry/bosh-agent/blob/9339c991d3abadfcc9e2e59fe4646f1f3440419a/infrastructure/http_metadata_service.go#L308)

test changes
I had to do quite a bit of updating for the setup integration tests.
The background is that once the registry related tests ran, they rewrote
the original json [file](https://github.com/cloudfoundry/bosh-agent/blob/f974e36207e2105082f1ecf935f64974583a3283/integration/assets/config-drive-agent.json#L12)s in /var/vcap/bosh and starting an agent without
the registry failed. Instead of making it work with the registry I opted
to take a backup of the original settings instead. The reasoning is that
we will soon remove all the registry code anyway and will have to rewrite
the integration tests without requiring the registry for setups.

To make sure that the backup of the original settings (from the time the vm
is deployed) is only run once, I switched the BeforeSuite to SynchronizedBeforeSuite.
When writing and testing the test I noticed that in some cases the agent.json
in /var/vcap/bosh was overwritten with the contents of this file

Additionally this PR updates the RunCommand method to actually
return stderr if a command fails so we can check for expected linux
error output.

Tracker Story: #180698114

#268

on warden the nats fw feature breaks the agent start with

2022-03-15_22:58:56.43738 [main] 2022/03/15 22:58:56 ERROR - Agent exited with error: Running bootstrap: Setting up networking: Setting up Nats Firewall: Error setting up cgroups for nats api access: cgroups: cgroup mountpoint does not exist

add check if cgroups are mounted to the if that handles early returns

@nouseforaname nouseforaname changed the base branch from main to 2.268.x March 23, 2022 17:09
Copy link
Contributor

@rkoster rkoster left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Add code to the agent to secure the remote mbus that
bosh provides. Provide a mechanism to restrict workloads
on the VM other than the agent itself to connect to BOSHs
mbus.

On VM creation bosh provides the Agent on the VM with creds
to connect to Nats. Worklaoads that live on the vm could
extract these credentials and use them to connect to nats and
extract further credentials that are being sent over nats once
the rendering of jobs takes place.

This creates a FW rule within the agent bootstrap to limit which
binaries can create outgoing requests to the nats endpoint.

The created FW rules are ephemeral and exist as long as the agent
process runs. This avoids the necessity for cleaning up FW rules if
the remote address of the nats should change. If a process starts
before the rule exists, the rule will still apply once it exists.

Updated tests relying on SetupNetwork calls with new method signature
this commit contains the linux counterpart for #259

This adds the setup firewall implementation based on containerds
cgroups and the go-iptables package. It is a replacement for the
stemcell based approach that didn't work reliably.

On VM creation bosh provides the Agent on the VM with creds
to connect to Nats. Worklaoads that live on the vm could
extract these credentials and use them to connect to nats and
extract further credentials that are being sent over nats once
the rendering of jobs takes place.

It returns early if the mbus url is an empty string:
- e.g. http_metadata_service calls SetupNetworking
before we can possibly have the mbus url because it is
responsible for extracting the agent settings.json from the
metadata endpoint.

or if it mbus url starts with https://:
- on a create-env we will see the mbus URL start with https as this
is where the cli posts the data via a tunnel. In that case we do
not want a firewall rule because the ssh tunnel the bosh-cli uses
will have the traffic blocked (since the ssh session wouldn't be
part of the cgroup) and therefore the create-env will fail because
the cli cannot send msgs.

the rest of the code
- sets up the iptables rules to block outgoing traffic to the mbus
  endpoint found in agent settings.json based on cgroup membership
- sets up the cgroup and adds it's own PID to the created cgroup.

Additional changes:
- updated the RunCommand to actually include stderr if the command
  on the target host fails
- added unit tests for return early behaviour
- added integration test for the created gw rules setup
- updated windows implementation to also return early
Tracker Stories #180698114 #181114075
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

None yet

2 participants