New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: Scaleway builders offline #21237

Open
bradfitz opened this Issue Jul 31, 2017 · 15 comments

Comments

Projects
None yet
7 participants
@bradfitz
Member

bradfitz commented Jul 31, 2017

Our scaleway account has been locked for abuse.

Our 50 linux-arm builders are offline.

/cc @jessfraz @kevinburke @adams-sarah @cybrcodr @aclements @randall77

@gopherbot gopherbot added this to the Unreleased milestone Jul 31, 2017

@gopherbot gopherbot added the Builders label Jul 31, 2017

@kevinburke

This comment has been minimized.

Contributor

kevinburke commented Jul 31, 2017

In hindsight, maybe BenchmarkBitcoinHash(b *testing.B) was not the best idea

@gopherbot

This comment has been minimized.

gopherbot commented Jul 31, 2017

Change https://golang.org/cl/52130 mentions this issue: dashboard: remove linux-arm-scaleway while we fix issues

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jul 31, 2017

@kevinburke, we still don't know what "abuse" they're talking about.

I opened a ticket to ask. They have an "Abuses" page to list abuse reports open against you, but there's nothing there:

screen shot 2017-07-31 at 9 07 38 am

screen shot 2017-07-31 at 9 07 49 am

gopherbot pushed a commit to golang/build that referenced this issue Jul 31, 2017

dashboard: remove linux-arm-scaleway while we fix issues
Updates golang/go#21237

Change-Id: Iae62120b96235fae84d6c689802506daeac45ca8
Reviewed-on: https://go-review.googlesource.com/52130
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@bradfitz

This comment has been minimized.

Member

bradfitz commented Jul 31, 2017

Scaleway replied:

One of your server (163.172.128.145) was detected being part of a DDoS targeting 81.30.144.118/32.
Here is the log of the detection : DDOS from IP 163.172.128.145 (attack ID 640640): protocols : tcp, targets: 81.30.144.118/32, sports: Dynamic (1024-65535), dports: 58823
Your account was automatically locked 3 days ago, this should automatically have opened a ticket 3 days ago to let you know about the issue but it didn't.
This apparently didn't and your server were automatically suspended after 48 hours as you obviously didn't answer to the ticket.

We will further investigate about the root cause to understand why you didn't receive any notification.

We sincerely apologise for this issue and we've added a 20€ discount on your account.

Please make sure no server was hacked in your fleet.

I replied:

We had too many tickets open, so I had to close some old ones in order to open this ticket.

Perhaps your automated system to open a ticket to tell us about abuse also failed to open a new ticket due to the ticket limit.

I'm very surprised that we'd be part of a DDOS. Our images are pretty recent (Xenial) and only listen on port 22 (ssh). I'm not aware of any recent OpenSSH vulnerability since Xenial's time.

But we'll investigate.

Thanks.

So, open questions, assuming our instances were actually 0wned:

  • are our Xenial listening on anything besides port 22?
  • are there recent OpenSSH vulnerabilities since Xenial?
  • should we use the new Scaleway firewall feature to restrict network access from a trusted IP/range?

I leave this to @jessfraz and @adams-sarah.

@gopherbot

This comment has been minimized.

gopherbot commented Jul 31, 2017

Change https://golang.org/cl/52192 mentions this issue: dashboard: turn on arm-scaleway builders

@jessfraz

This comment has been minimized.

Contributor

jessfraz commented Jul 31, 2017

nmap says only ssh is open, but I am doing a more full depth scan for due-diligence

@adams-sarah

This comment has been minimized.

Contributor

adams-sarah commented Jul 31, 2017

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jul 31, 2017

firewall sounds like a good idea. no harm, anyway.

Well, there is harm: we have to maintain a bastion host, and we have the inconvenience of having to jump through it or VPN through it or copy files through it whenever we're trying to work. It's not completely free.

gopherbot pushed a commit to golang/build that referenced this issue Jul 31, 2017

dashboard: turn on arm-scaleway builders
Updates golang/go#21237

Change-Id: Iaaa2f03543d9b85de5bd30814aecacb6d85b8a66
Reviewed-on: https://go-review.googlesource.com/52192
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@jessfraz

This comment has been minimized.

Contributor

jessfraz commented Jul 31, 2017

I turned the scaleway builders back on they should work now.

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jul 31, 2017

I logged into one of our now-back-up 50 ARM servers to check the image's OpenSSH version:

# dpkg -s openssh-server | grep ^Version
Version: 1:7.2p2-4ubuntu2.1

https://packages.ubuntu.com/xenial/openssh-server says 7.2p2-4ubuntu2.1 is the latest.

So, I'm starting to doubt the whole DDoS thing.

@adams-sarah

This comment has been minimized.

Contributor

adams-sarah commented Jul 31, 2017

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jul 31, 2017

I think Scaleway's mapping from (time, source IP) => customer is flawed. But I also haven't logged the start/stop time of each machine's ephemeral IP addresses either, so I can't say. And they also didn't tell us a time.

@netroby

This comment has been minimized.

netroby commented Jul 31, 2017

you should not put all server open to public. you can have 1 or 3 jump gateway host. then make other only have private acess via private ip/net range. then add firewall for jump host limit acess ip.

Sent from my Xiaomi Mi Note 2 using FastHub

@bradfitz

This comment has been minimized.

Member

bradfitz commented Jul 31, 2017

@netroby, yeah, that's what the "bastion host" referred to above is.

Like I said, there's a non-zero cost in maintaining that, and the machines we're defending against are stateless, can be reimaged & rebooted per builds, and are not valuable (open source code only), so the cost may not outweigh the benefits. It might be easier to just regularly rebuild the images we're using if there are OpenSSH exploits. But we've been running the latest Xenial LTS code with security updates, which is why I suspect Scaleway's finger pointing at our instance is flawed. I doubt somebody wasted their 0 day exploits on the Go continuous build system to do a DDoS attack.

@jhelbling

This comment has been minimized.

jhelbling commented Aug 1, 2017

  • Use different port for SSH
  • Us Firewall function from Scaleway. Very simple and if you have problems - it is scaleway support job.
  • Use old function aka 'white listing'
  • Use tunnels
  • Use simple protection tools aka fail2ban or what ever.

Cheers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment