Servers

[Changaco]

It's time to reconsider the technical infrastructure that the liberapay.com website runs on.

So far Liberapay has always been hosted on AWS, first indirectly through OpenShift Online (#1), then directly (#177). However, it's becoming more difficult to use AWS without getting a somewhat large bill. Moreover, some flaws of AWS have become apparent in the last two years, for example the Apache failures which regularly result in downtime. Technically, we could switch to nginx or gunicorn without leaving AWS, but if we have to build the stack ourselves then we might be better off doing it completely and on cheaper servers.

Liberapay has also always been served through Cloudflare, because we've always needed CNAME flattening (#1 (comment)). I'm not in any hurry to stop using Cloudflare, it's a pretty good service, but it's part of our infrastructure so it's relevant to this issue. If we migrate away from AWS, then we have to decide whether we want to pay Cloudflare and use its load balancer, or build our own.

The overall dilemma is mostly the same it was 3 years ago: doing things ourselves gives us more control and can reduce monetary costs, but it requires more work and can therefore slow down the progress on improving other aspects of Liberapay. However, considering the time I've spent trying to understand AWS and configure it, I'm not sure how much time it really saves me compared to building and maintaining our own stack. Moreover, Liberapay has changed a lot in the last 3 years, so perhaps it now needs better infrastructure as much as it needs improvements in other areas.

[Codeberg-org]

Is there some overview about the currently used tools/services/setup that are required to operate liberapay (potentially to share offline if this information is sensitive? If necessary please contact us directly to discuss ideas: codeberg at codeberg.org, please see https://codeberg.org and https://pages.codeberg.org for an overview of what we already do).

[Changaco]

@Codeberg-org Liberapay is quite simple, it's a Python web app connected to a PostgreSQL database. Does that answer your question?

[Codeberg-org]

@Changaco : this does not sound too hard. Presumably some reverse proxy in front and guessing from #1728 also SMTP?

[Changaco]

Getting the Liberapay web app to run isn't hard, but if we're going to migrate away from AWS, then I want the new stack to be at least as good as the current one, and better in at least one way, otherwise it's not worth it. Here are some things that come to mind:

• zero downtime architecture (at least two machines, in separate data centers)
• automated weekly updates (preferably everything from the kernel up, not just security updates like on AWS)
• metrics dashboard (similar to AWS CloudWatch, but with RAM usage)
• logs aggregation (again, similar to AWS CloudWatch)
• log-based alerts (AWS is currently configured to send me an email every time the average response time of liberapay.com goes too high)
• extensive audit logs (ideally all commands and SSH sessions should be logged, we currently don't have that).

Presumably some reverse proxy in front

As mentioned in my first message, we would have to decide whether to rely on Cloudflare, in which case our servers wouldn't need to listen for HTTP traffic from the Internet at all (#1093), or to keep our infrastructure more independent from Cloudflare.

guessing from #1728 also SMTP?

A traditional SMTP daemon is of no use. The software we need for #1728 doesn't exist yet, so at first we would probably send emails directly from the Liberapay web app, which is something that we can't do now because the AWS EC2 instances don't have stable IP addresses.

[Urahara]

Today Liberapay is based on t2 instances and Postgres rds? It's not worth migrate to EKS or t3 instance (is more cheap)?

[Changaco]

@Urahara You're right, it looks like t3.micro instances are 9.5% cheaper than the t2.micro instances we're currently using, even though t3.micro has 2 vCPUs whereas t2.micro has only one, according to https://aws.amazon.com/ec2/pricing/on-demand/ (Ireland region).

[Changaco]

The t3a.micro instances (variants based on AMD EPYC processors) are even cheaper, so apparently we can save 19% simply by switching from t2.micro to t3a.micro.

[Changaco]

I've switched to t3a.micro, however most of our EC2 bill is for the Application Load Balancer, not the instances that actually run the Liberapay web app, so the savings will be small.

[Changaco]

(AWS' annoyingly complex pricing is one of the reasons why I would like to migrate away from it.)

[Changaco]

which is something that we can't do now because the AWS EC2 instances don't have stable IP addresses.

We could probably change that by allocating an Elastic IP address, however it currently only works for IPv4.

[gileri]

most of our EC2 bill is for the Application Load Balancer

If you only use ALB for healthchecks/failover and use static listening ports on application servers (i.e. port 80/443), you may be able replace it with AWS Route53 weighted records and failover.

That would bring the monthly cost of failover from ~20$ to 2.5$ :

• $0.50 for 1 DNS zone
• ~$2 for DNS queries based on 100 CDN servers (Cloudflare) servers and a DNS record TTL of 60 seconds
• $0 for the healthchecks, up to 50 servers checked.

AWS' annoyingly complex pricing is one of the reasons why I would like to migrate away from it

I may see why 😅

[TheEvilSkeleton]

Any update?

[Changaco]

So far I've only looked into which hosting provider would be best for Liberapay. There are a lot of candidates…

One way to eliminate many of them is to keep only the ones that host servers in several countries, since most providers are small and local. We could technically use multiple providers if we were ready to have servers around the world, but it's simpler to use a single provider.

The list of “global” providers includes Vultr, Linode, and DigitalOcean.

Ideally Liberapay would have servers in at least France and either South Korea or Japan. France is an obvious choice since it's the legal home of Liberapay and has one of the cleanest electricity supply in the world. South Korea and Japan have good trade relationships with the EU, and having servers there could reduce latency for Liberapay users in that part of the world.

[Urahara]

@Changaco What about Scaleway? Is look like they have a good price.

[Changaco]

@gileri Thanks for the suggestion, but the load balancer is automatically provisioned by Elastic Beanstalk, not by us, and I don't think we can tell EB not to use load balancers.

[Changaco]

@Urahara I didn't mention Scaleway because it's not a global provider, it only hosts server in France and the Netherlands.

[TheEvilSkeleton]

@Changaco I've heard about OpenStack, which is FOSS: https://www.openstack.org/

[gileri]

@Changaco

Ideally Liberapay would have servers in at least France and either South Korea or Japan [...] and having servers there could reduce latency for Liberapay users in that part of the world.

Are CDNs (and their dedicated/optimized links) not sufficient to obtain a reasonable latency in Liberapay ?

Even if application servers are hosted near users, the database would surely remain centralized. Meaning that round-trips across the world would still be required for distant users.
Or you are thinking of switching to a global (multi-region) database ?

[Changaco]

Are CDNs (and their dedicated/optimized links) not sufficient to obtain a reasonable latency in Liberapay ?

I don't know, we would have to ask our users in Asia if they perceive the network latency when browsing Liberapay.

Even if application servers are hosted near users, the database would surely remain centralized.

Each web server would need its own copy of the database, otherwise latency would be increased instead of lowered. However, the replicas could be read-only and send write queries to a primary.

To clarify: having servers around the world isn't a priority and isn't a requirement to close this issue, it's just a possibility that should be kept in mind in order to avoid making choices that would complicate spreading servers geographically in the future.

[Changaco]

@TheEvilSkeleton I'm aware of OpenStack, but I've never used it.

[TheEvilSkeleton]

That might be a great place to do it, since OpenStack is FOSS, and LiberaPay is FOSS as well.

[revi]

Are CDNs (and their dedicated/optimized links) not sufficient to obtain a reasonable latency in Liberapay ?

I don't know, we would have to ask our users in Asia if they perceive the network latency when browsing Liberapay.

At least I am being routed to cf-ray | [REDACTED]-NRT - Japan. (Cloudflare doesn't route Korean users to ICN server because traffic cost here is insane.[EDIT/NOTE: Article is dated 2016 but the rules stay the same so there's no change for crazy pricing])

According to Firefox, it takes about 4.5 seconds to load https://liberapay.com/explore/organizations (random page with more than a few images) without any cache, and I take it as a compromise for connecting to a server traveling about the opposite side of the earth, but opinions might vary.

[gileri]

Thank you for the datapoint @revi.
In France, from a 1Gbps FTTH connection on a desktop it takes me 1 second to load https://liberapay.com/explore/organizations.

[Changaco]

Measuring the total load time of an /explore/… page mostly measures how long it takes to load the avatars from various services, not the time it takes to load the HTML from the Liberapay web server. The latter is the only load time that could be lowered if we had multiple servers around the world instead of a single server in Europe.

Measuring the load time of https://liberapay.com/about is more relevant. In a Firefox that already has the static assets in cache, I get: 9 requests | 221.45 KB / 4.50 KB transferred | Finish: 436 ms | DOMContentLoaded: 336 ms | load: 366 ms.

[revi]

GET /about/
11 requests
540.01 KB / 219.48 KB transferred
Finish: 1.79 s
DOMContentLoaded: 1.52 s
load: 1.75 s

FYI: Additionally my profile and liberapay team profile because that's where people most likely interact.

GET /revi/
15 requests
596.29 KB / 279.69 KB transferred
Finish: 1.90 s
DOMContentLoaded: 1.52 s
load: 2.64 s

GET /Liberapay/
53 requests
1.95 MB / 1.66 MB transferred
Finish: 5.33 s
DOMContentLoaded: 2.67 s
load: 5.82 s

[gileri]

GET /about/
16 requests
729,35 Ko / 412,91 Ko transferred
Finish : 628 ms
DOMContentLoaded: 382 ms
load: 479 ms

GET /revi/
21 requests
866,78 Ko / 556,70 Ko transferred
Finish : 503 ms
DOMContentLoaded: 279 ms
load: 502 ms

GET /Liberapay/
58 requests
2,30 Mo / 2,01 Mo transferred
Finish : 2,63 s
DOMContentLoaded: 670 ms
load: 2,63 s

[everdha]

Hi there, Liberapay being the only ethical and libre support/donation platform out there (thank you <3), it is important for this community, (and also under the lens of security and surveillance issues) to go away from AWS. There are alternatives nowadays. Here are some experienced suggestions :

• Digital Ocean offers infrastructure (servers etc) for several open source projects, hacker spaces etc, contact them and I am sure they will propose something (see https://www.digitalocean.com/community for some of their sponsoring, but better directly write to them for your sepcific needs)
• Gandi supports open projects also (with free hosting and servers, not sure they have other then iaas but maybe they support also in other ways, they are also very nice, do not hesitate to contact them).
• Les CHATONS, the libre/ethical french net of service providers I am sure will be able to give you advise and better recommandations, on their forum https://forum.chatons.org. I know they like Liberapay <3, I also know they want to collaborate. But that is another issue I will write about :)

[everdha]

Privacytools.io maintains a list of secure and open source email providers here : https://www.privacytools.io/providers/email/
Tutanoto, which is also on their list, offers discounts on premium accounts for organisations : https://tutanota.com/fr/discount/

[TheEvilSkeleton]

Keep in mind that while Tutanota's clients are open source, the server side is proprietary.

[everdha]

Keep in mind that while Tutanota's clients are open source, the server side is proprietary.

Yes indeed, ans same for ProtonMail by the way (first in the list of privacytools.io). Thanks for reminding this.

[Changaco]

Lesson from #2039: cloudflared is unreliable. We need at least one of:

• another implementation of cloudflare tunnels, preferably as a library that can be used directly by the Python web server (gunicorn)
• our own ready-to-use reverse proxy to fall back to in case tunnels are broken

[Changaco]

Liberapay was down for 7 hours because cloudflared stopped working again for no apparent reason.

[fschrempf]

If you decide to build your own hosting stack without any of the "big" providers, I can recommend Hetzner. They offer VPS instances with good price/value based in Germany, Finland and the US. I think it would meet most of the requirements discussed above.

I have only used Hetzner for smaller setups/projects so far, but I know that other FOSS-, privacy-minded businesses/orgs use it successfully (e. g. Plausible).

[Coehill]

I'd like to see LiberaPay move away from cloudflare personally, since it's a privacy nightmare.

Hetzner is a solid alternative to AWS too.