debuilder.infra.ooni.io compromised month ago

[darkk]

Impact: RCE on a build host unnoticed for a month

Detection: manual observation of anomalies on a broken box

Timeline UTC:
02 Jul 23:41: start time of payload running under jenkins uid
09 Jul 07:43: first No space left on device in the series Jul-09...Jul-23
04 Aug 18:17: dom0-bootstrap.yml failed during apt update: 100% disk space used, 93% inodes used (logrotate is broken since January)
04 Aug 18:45: incident published
04 Aug 19:20: debuilder.infra.ooni.io turned off

What went wrong:

• Seems, it was CVE-2017-1000353, jenkins RCE before login

What could be done to prevent relapse and decrease impact:

• network-level authentication (SSH, VPN, nginx with basic auth) on internal services
• logrotate monitoring
• alert on high usage of CPU (miners) or NET (DDoS bots)
• alert on SMTP traffic

[darkk]

network-level authentication (SSH, VPN, nginx with basic auth) on internal services

I had a bikeshedding session with @hellais about that and we concluded:

• nginx with basic auth stores secret in browser memory till reboot and that's OK
• nginx with basic auth has ugly login form and that's OK
• nginx with basic auth has no 2FA / U2F support and will never have and that's OK
• nginx with basic auth requires good random non-memorable passwords to make pw check single round of SHA1 and that's OK
• nginx with basic auth requires separate password for every single webservice and that's OK as everyone knows how to use password manager
• SSO is nice and nginx-sso following pubcookie model is quite simple, but that's still hundreds lines of golang code and some extra complexity and potential SPOF

[darkk]

Awesome example of in-app auth bypass:
https://bo0om.ru/just-enter-the-space-attacks
Sorry, in Russian. Long story short: just add %20, authentication and url routing are done in wrong order – boom!

[hellais]

ooni/backend#343