New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade ossec to v3.0.0 #3724

Merged
merged 5 commits into from Oct 3, 2018

Conversation

Projects
None yet
4 participants
@emkll
Copy link
Contributor

emkll commented Aug 21, 2018

Status

Ready for review

Fixes #3701:

Updates ossec to v3.0.0, modifies build logic to use GPG signatures for verifying source.

Testing

Clean install scenario

  • make build-debs
  • Provision staging vms on this branch
  • ./var/ossec/bin/ossec-control status returns running for all except ossec-execd
  • Observe ./var/ossec/bin/list-agents -c on mon server returns the app agent
  • Observe that ossec mails are sent if email is configured is prod

Upgrade scenario

  • make build-debs
  • copy and install securedrop-ossec-* and ossec-* to app and mon servers
  • ./var/ossec/bin/ossec-control status returns running for all except ossec-execd
  • observe ./var/ossec/bin/list-agents -c on mon server returns the app agent
  • Observe that ossec mails are sent if email is configured is prod

Deployment

Updates to securedrop-ossec-{agent,server} and ossec-{agent,server} will be served by apt and will be upgraded automatically.

Checklist

If you made changes to the server application code:

  • Linting (make ci-lint) and tests (make -C securedrop test) pass in the development container

If you made changes to securedrop-admin:

  • Linting and tests (make -C admin test) pass in the admin development container

If you made changes to the system configuration:

If you made non-trivial code changes:

  • I have written a test plan and validated it for this PR

If you made changes to documentation:

  • Doc linting (make docs-lint) passed locally

@emkll emkll requested review from conorsch and msheiny as code owners Aug 21, 2018

@emkll emkll referenced this pull request Aug 21, 2018

Closed

Update OSSEC to v3.0 #3701

@emkll emkll force-pushed the 3701-bump-ossec-3.0.0 branch 3 times, most recently from 76ec489 to 146b813 Sep 7, 2018

@emkll emkll force-pushed the 3701-bump-ossec-3.0.0 branch from 146b813 to f7eb0d3 Sep 28, 2018

@emkll emkll changed the title WIP - Do not merge - Upgrade ossec to v3.0.0 Upgrade ossec to v3.0.0 Sep 28, 2018

@emkll emkll force-pushed the 3701-bump-ossec-3.0.0 branch from f7eb0d3 to f531d03 Sep 28, 2018

@msheiny

This comment has been minimized.

Copy link
Contributor

msheiny commented Sep 28, 2018

So far I'm testing on an upgrade scenario path...

I'm actually installing from a 0.9.1 stock.. and then i ran the apt install of the latest ossec packages on both servers... Seems like mon comes up fine but app showed this status

root@app-staging:/home/vagrant# /etc/init.d/ossec status
ossec-logcollector: Process 5822 not used by ossec, removing ..
ossec-logcollector not running...
ossec-syscheckd not running...
ossec-agentd: Process 5818 not used by ossec, removing ..
ossec-agentd not running...
ossec-execd not running..

i tried to reboot that service and i get

root@app-staging:/home/vagrant# /etc/init.d/ossec restart
ossec-logcollector not running ..
ossec-syscheckd not running ..
ossec-agentd not running ..
ossec-execd not running ..
OSSEC HIDS v3.0.0 Stopped
Starting OSSEC HIDS v3.0.0 (by Trend Micro Inc.)...
Started ossec-execd...
2018/09/28 22:15:42 ossec-agentd: INFO: Using notify time: 600 and max time to reconnect: 1800
Started ossec-agentd...
Started ossec-logcollector...
2018/09/28 22:15:45 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2018/09/28 22:15:45 rootcheck(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2018/09/28 22:15:53 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2018/09/28 22:15:53 rootcheck(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2018/09/28 22:16:06 ossec-syscheckd(1210): ERROR: Queue '/var/ossec/queue/ossec/queue' not accessible: 'Connection refused'.
2018/09/28 22:16:06 rootcheck(1211): ERROR: Unable to access queue: '/var/ossec/queue/ossec/queue'. Giving up..
ossec-syscheckd did not start

Ya want to sync on this next week? I'd love to take a more active role in ushering this ticket thru if ya want some help. Maybe I just did something silly though -- so I'm also happy to get yelled at next week :)

@msheiny msheiny force-pushed the 3701-bump-ossec-3.0.0 branch from f531d03 to 6a562fd Oct 1, 2018

@msheiny

This comment has been minimized.

Copy link
Contributor

msheiny commented Oct 1, 2018

rebased on develop 71ca2ed58

@emkll

This comment has been minimized.

Copy link
Contributor

emkll commented Oct 2, 2018

After spending some time debugging, it seems like the agent and server debs that are compiled with the build-ossec-deb-pkg role (https://github.com/freedomofpress/securedrop/blob/develop/install_files/ansible-base/roles/build-ossec-deb-pkg/tasks/main.yml#L95) now provide an empty /var/ossec/etc/client.key, which causes the ossec agent to fail startup. This was not the case with debs build from upstream, nor with version 2.8.2 debs.

This is also why the clean install works, but the upgrade scenario does not, as the client key is overwritten in the configuration.

@emkll

This comment has been minimized.

Copy link
Contributor

emkll commented Oct 2, 2018

@conorsch do you have any clue as to where the contents of /var/ossec/ comes from on this line ?https://github.com/freedomofpress/securedrop/blob/develop/install_files/ansible-base/roles/build-ossec-deb-pkg/tasks/main.yml#L95 ? The contents of this are likely overwriting the client.keys

EDIT: Not sure how I missed it, but https://github.com/freedomofpress/securedrop/blob/develop/install_files/ansible-base/roles/build-ossec-deb-pkg/tasks/main.yml#L84 installs ossec on the build host and then contents of /var/ossec are moved into the build dir. Removing the empty client.keys appears to do the trick !

@emkll emkll force-pushed the 3701-bump-ossec-3.0.0 branch 2 times, most recently from 27ac0f0 to b8f97d2 Oct 2, 2018

emkll added some commits Aug 21, 2018

Bump ossec version strings to 3.0.0
Ossec 3.0.0 contains several bug and security fixes, as such we should
upgrade securedrop ossec agents and servers to this release.
Use GPG to verify ossec tarballs
As of 2.9.3, ossec source tarballs are now signed with a GPG key,
instead of providing checksums. The following should now verify source
code tarballs when packages are built/
Update ossec registration logic for 3.0
- Generate and use shared secret which is required for agent registration
- agent-auth now returns 0 when registration failure occurs
Replace mon server alias and smtp_server with ip
Due to issues with ossec 2.8.2+ and disabling of the ipv6 stack, name
lookups can't `getaddrinfo: Name or service not known`. Using ip
addresses in lieu of aliases sidesteps the issue.

- Since ossec.conf is not templated, securedrop-ossec agent and server will replace
these values as part of the postinst.
Remove client.keys
An empty client.keys was overwriting /var/ossec/etc/client.keys with an empty one, breaking the registration between client and server. Removing the empty client.keys prior to building will ensure the keys are preserved during an upgrade.

@emkll emkll force-pushed the 3701-bump-ossec-3.0.0 branch from b8f97d2 to bddc30d Oct 2, 2018

@conorsch

This comment has been minimized.

Copy link
Contributor

conorsch commented Oct 3, 2018

installs ossec on the build host and then contents of /var/ossec are moved into the build dir

That's right, and it's specifically the USER_DIR value in the vars template that sets /var/ossec as the dest.

Removing the empty client.keys appears to do the trick !

Glad to hear it! Given the cycles we're spending on pulling off the upgrade, now seems a fine time to bump mention of #2140, in case it would reduce our maintenance burden.

@emkll

This comment has been minimized.

Copy link
Contributor

emkll commented Oct 3, 2018

Thanks for the help @msheiny, this is ready for re-review: based on my testing in prod VMs the bug you experienced should be fixed, emails now flow in.
Since the existing postinst scrips work with 3.0.0, I've opted for the smallest diff for this PR, perhaps we should maybe revisit the build strategy and/or using wazuh in an upcoming engineering meeting.

@zenmonkeykstop

This comment has been minimized.

Copy link
Contributor

zenmonkeykstop commented Oct 3, 2018

@emkll is there much if any change so far to the user experience (email formatting, sensitivity of triggers etc) compared to 2.8.2?

@emkll

This comment has been minimized.

Copy link
Contributor

emkll commented Oct 3, 2018

@zenmonkeykstop: good question. The emails look the same, and based on < 24h of observations, the alerts appear to be the same as before, but we should keep an eye out for new alerts and update the docs if necessary. Release notes state that were some changes in 2.8.3 for Apache/Postfix, but a closer look at changes in https://github.com/ossec/ossec-hids/tree/master/etc/rules indicates they are minor.

@msheiny
Copy link
Contributor

msheiny left a comment

Looking good on a second pass of an upgrade test! 🎉

Confirmed services were up and running on 3.x on both servers.. connection maintained .. and alerts were being processed by tailing the logs. I also rebooted boxes and confirmed behavior remained.

Doing scrach test now.

@msheiny

msheiny approved these changes Oct 3, 2018

Copy link
Contributor

msheiny left a comment

Yep looking good on the from scratch front as well! Hooraaaaaayyyyyyyyyyyyyyyyy

@msheiny msheiny merged commit bcc624d into develop Oct 3, 2018

5 checks passed

ci/circleci: admin-tests Your tests passed on CircleCI!
Details
ci/circleci: lint Your tests passed on CircleCI!
Details
ci/circleci: staging-test-with-rebase Your tests passed on CircleCI!
Details
ci/circleci: tests Your tests passed on CircleCI!
Details
ci/circleci: updater-gui-tests Your tests passed on CircleCI!
Details

@msheiny msheiny deleted the 3701-bump-ossec-3.0.0 branch Oct 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment