Implementation Discussion Thread: Backend

[Gaelan]

Here's a thread to discuss the implementation of the backend, including mod hosting and the API. Questions we need answered before we can start work on this:

• What technologies will be used to implement this?
• Where will this be hosted? How will we cover the costs, if any, for said hosting?
• Who will work on this?

[Gaelan]

Initial thoughts:

• Mods and maybe metadata in S3
  • Use S3's BitTorrent support to save us on download bills for public mods. We would need to prompt about seeding on the first install; this should be a yes/no prompt, because a checkbox with next will either get people with data caps to accidentally blow through them or make people who would be willing to seed not do so because they missed the option.
  • If we store metadata here, then downloading public mods and modpacks (the most common use case) doesn't use any server resources.
• Writes, private reads via AWS Lambda
  • Infinitely scalable, very cheap
  • Private mod downloads would provide a token which can be used with S3 for downloads. Sadly, we can't bit torrent these.
• Is a DB necessary? Our data might be simple enough for JSON in S3.

This certainly isn't perfect, but I hope it's good enough to start a discussion.

[MaPePeR]

Out data might be simple enough for a Json on GitHub, pointing to other locations.

[Gaelan]

@MaPePeR That's also an option, though modpacks need to be build able without JSON tinkering. Also, I feel like we're abusing GitHub here; also we'll end up with a HUGE repository.

On a related note, if mods are already hosting releases somewhere neutral (i.e. not Curse, maybe GH Releases), we should provide an option for that. S3 isn't cheap.

[DavideRoss]

I'm really like AWS Lambda, I can write handlers in Node.js or Python. About the database I think it's better to have one, so in the future we can provide analytics to modders/modpack creators/users.

[Gaelan]

@DavideRoss Perhaps. What would you suggest using?

[DavideRoss]

@Gaelan currently I'm using MongoDB for almost every project, but also ElasticSearch or DynamoDB are good alternatives. I excluded MySQL and PostgreSQL because they are pretty heavy.

[Gaelan]

SQL has its advantages, namely performance without needing to manage denormalization.

[DavideRoss]

It's fine to me, I have to refresh my knowledge but it's ok.

[rchard2scout]

AWS Lambda sounds like a good start, but it limits the possibilities a little in terms of programming languages...

• Python: I don't know if it'll be fast enough, but it'll be easy to program
• Java: A little harder to program, but most people in the MC modding community have experience with it.
• Node.js: Javascript Is Not A Back-End Language. Oppose on principle, but I could be convinced with good arguments.

S3: Looks good for distribution. I don't know about the bittorrent part, for three reasons. The first is that it creates complications in the intellectual property/copyright area. The second is that some ISPs are blocking the bittorrent protocol because of piracy, and the third is that it'll further complicate our technology stack.

Database: Yes, definitely. From what I can think of so far, our data will be fully relational, so an SQL database sounds like the best idea. We'll have to design a database schema at some point.

[Gaelan]

Keep in mind that there is nothing forcing us to use the languages they provide--we could always to something like writing a small Node script to run a Ruby script with an interpreter included in our bundle. Also, pretty much everything runs on the JVM.

EDIT: answer to others
We will need to watch our S3 bill. $1 will allow us to distribute 600 copies of a large modpack (I got the file size from IE:S, but other packs are probably similar).

I'm with you on SQL. Where should we host it?

[Gaelan]

Someone on Reddit mentioned http://nodecdn.net, which looks really cool. Free hosting for mod developers.

[MaPePeR]

Node script to run a Ruby script with an interpreter included in our bundle.

Please no...

[rchard2scout]

We could just use Amazon EC2. It'll require a bit more configuration, but that's not a problem. It's also very easily expandable once we need the power.

[DavideRoss]

As I said multiple times before, currently my skills are limited on Node.js and a bit of Python. I have some attitude on learning and I really enjoy it, but between the product I can deliver and the solid product that this project will be there is an abyss.

I'm currently using EC2 for a couple of projects, with CentOS and Node.js/nginx, if you need to do routine works on the machine I'm here. Otherwise I can work on website/GUI.

[rchard2scout]

My skills are mainly C++/C/Java, with a bit of SQL, PHP, Perl, and Javascript mixed in. I've also got experience with both CentOS and Ubuntu (not in EC2 though), so it looks like we'll be able to pull off the server management part.