New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get quotes on additional services from Equinix #231

Open
pnorman opened this Issue Aug 2, 2018 · 12 comments

Comments

Projects
None yet
4 participants
@pnorman
Copy link
Collaborator

pnorman commented Aug 2, 2018

There's been a few services we've discussed having from Equinix and Cogent, and now that we're moved in and handling load we can look at them. I'll get prices on

  • A storage locker. This would give us rack space and becomes important if we add more than about 8U of machines to what's there right now
  • Additional power so we can run the tiamat machines as a ceph cluster
  • A second network connection for redundancy
  • Prepaid smart hands packages

Is there anything else we might want that I could get a quote on?

@jburgess777

This comment has been minimized.

Copy link
Member

jburgess777 commented Aug 2, 2018

A 10Gbps link (or two). Possibly staying at 1Gbps commit but allowing a higher burst speed. If I remember correctly, the Cogent docs say they will only do link aggregation on 10G ports. The Cisco switch has 10Gbps SFP+ ports and I think @Firefishy even purchased 10Gbps-LR module(s) in anticipation that we might want to upgrade the link speed.

@jburgess777

This comment has been minimized.

Copy link
Member

jburgess777 commented Aug 2, 2018

Not directly Equinix related but I think @Firefishy mentioned we might want to begin casual conversations with other ISPs at the same location about peering with them. We might want to wait a bit longer so there is some data about how much data we source before we do this. In this discussion, the more bandwidth we send to the various ISPs the more interested they might be in peering directly with us. I am guessing that the bandwidth for the planet files and tiles are going to be the most significant things.

@pnorman

This comment has been minimized.

Copy link
Collaborator Author

pnorman commented Aug 3, 2018

Tiles are going to be largely coming from the CDN, so it's going to be the planet. We know from bytemark that there's a good chunk of bandwidth being used by AWS customers, and there are facilities to directly link with Amazon in the data centre.

Would any of this work without our own ASN + routers that can do BGP stuff?

@jburgess777

This comment has been minimized.

Copy link
Member

jburgess777 commented Aug 3, 2018

Tiles are going to be largely coming from the CDN, so it's going to be the planet.

True, but the tile CDN traffic distribution is under our control. There is 70Mbps from Orm going to the caches. Trogdor is a tile cache in the Netherlands and it peaks at 120Mbps, we could host our own local cache and serve a similar amount of traffic if we wanted.

Would any of this work without our own ASN + routers that can do BGP stuff?

It can be done without BGP but the approach would be different. If we assume for a moment that another ISP gave a similar setup Cogent: a fiber with some IPv4/6 addresses which can be used on it. We can connect that to our switch on a new VLAN. Add an additional VLAN interface to any server we want to use this connection and assign an address.

To handle the multiple default routes the machine then needs a second routing table. Add the new default route to this new table. Then add a routing rule to specify that traffic to from the new interface uses the second route table. Traffic to/from the original IP will still flow via the existing route. Traffic to/from the new IP goes via the new link. Here are some links showing some related information:
https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System
https://unix.stackexchange.com/questions/345862/is-it-possible-to-have-multiple-default-gateways-for-outbound-connections
https://serverfault.com/questions/823058/how-to-set-up-two-default-routes-in-linux

The next problem is that we then need to make the right people use the new IP to access when accessing this server. This probably needs something similar to our existing geo-dns setup: when a DNS request for the machine arrives from a client matching a certain list of subnets then we respond with the DNS of the second IP.

In the approach above, each server using multiple uplinks has to be configured to use them. In a BGP world this complexity is handled by the routers instead. There is also another option where NAT is used to give the appearance that the internal servers have different addresses from what they actually use.

@tomhughes

This comment has been minimized.

Copy link
Member

tomhughes commented Aug 3, 2018

Many of our machines already have multiple gateways, but that is metric based so that they fall back to going via the internal network if the external network is down.

You can't exactly choose a default route by what interface is being used though, because it's the choice of route which tells you which interface to use. It's a basic chicken/egg problem. What you can do if the interface has a different address is to use policy routing to choose the route based on the source address of an outgoing packet.

@pnorman

This comment has been minimized.

Copy link
Collaborator Author

pnorman commented Aug 3, 2018

We currently have 3 kW of power and, last I checked, were about 2.2 used. We can probably get 6.5 kW without a physical upgrade (32A 220V + safety margin). What should I get quoted, 4kW and 5kW. I believe the tiamats have 1.4kW supplies, but I'm not sure what they actually draw.

If we want the tiamats there, it might turn out to be cheaper to replace older machines and upgrade the power by a lesser amount, but the incremental cost per kW will tell us that.

@jburgess777

This comment has been minimized.

Copy link
Member

jburgess777 commented Aug 3, 2018

For power: our original contract gave a per KVA cost per month so we can probably make a reasonable guess at what we would be charged (I won't repeat the value here in case it is considered sensitive). Any quote would likely only be valid for 30 days so unless we are reasonably sure we want to go ahead it might not make sense to ask for a quote right now.

The biggest unknown is whether they turn around and say they can't increase the capacity in this particular rack for some reason (like cooling or UPS capacity).

@tomhughes tomhughes added the equinix label Aug 5, 2018

@pnorman

This comment has been minimized.

Copy link
Collaborator Author

pnorman commented Aug 5, 2018

Email sent to our Equinix sales person for the ones that fall under them.

@pnorman pnorman self-assigned this Dec 8, 2018

@pnorman

This comment has been minimized.

Copy link
Collaborator Author

pnorman commented Dec 11, 2018

We're now peaking slightly over 3 kW.

image

How much do we want to go for? We probably have at least another 500W of machines we could turn on, so at least 4 kW I think. Perhaps go right for 6 kW?

@gravitystorm

This comment has been minimized.

Copy link
Collaborator

gravitystorm commented Dec 12, 2018

What's the consequence of going for 4 now, and 6 when we need it? Is there a setup charge that we'd need to pay twice? Or can we just upgrade when we need to?

@pnorman

This comment has been minimized.

Copy link
Collaborator Author

pnorman commented Dec 12, 2018

What's the consequence of going for 4 now, and 6 when we need it? Is there a setup charge that we'd need to pay twice? Or can we just upgrade when we need to?

I don't know about charges without getting the quote. There's probably some non-recurring charge for changing associated with the admin work, plus the admin work on our side.

Equinix doesn't guarantee that we'll be able to upgrade power in the future, it depends on availability at the location. I think AM6 has plenty of spare capacity right now, but I know of data centres where the primary limitation is power, not space.

If we think we'll use 6 kW in the medium future for Ceph, I'd rather go for it now. If Ceph is still a blue-sky project, then 4 kW would make sense.

@pnorman

This comment has been minimized.

Copy link
Collaborator Author

pnorman commented Jan 11, 2019

Status updates. The sales contact was on holiday like everyone else and nothing here was urgent, so I'll remind them on Monday.

A storage locker. This would give us rack space and becomes important if we add more than about 8U of machines to what's there right now

I've sent the pricing internally. I'm checking if the lockers are deep enough for a server sideways, or if it'd need to go on end.

Additional power so we can run the tiamat machines as a ceph cluster

With more machines turned on, it's more urgent, so I'm having them quote 4 kVA and 6 kVA.

A second network connection for redundancy

Sending the details around now

Prepaid smart hands packages

This is not worth it right now. We can revisit when we start using remote hands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment