Skip to content

Backend Architectures

Jump to bottom Edit New page
Jimmy Moon edited this page Jun 22, 2013 · 16 revisions

Backend Architectures

Twitter

ror, scala, jetty, erlang, thrift, mongrel, comet server, my-sql, memchached, varnish, kestrel(mq), starling, gizzard, cassandra, hadoop, vertica, munin, nagios, awstats

Facebook

php (with hiphop compiler), thrift, java(tomcat, jetty, minor),epoll, erlang, tornado, nodejs my-sql, memcahced, hadoop, hbase, hive, scribe(-hdfs), bigpipe, varnish, haystack, cassandra

Tumblr (Korean)

centos, sciapache, apache, nginx, (move out of)php, scala(selection), ruby, thrift, my-sql, redis, hbase, memcached, gearman, kafka, kestrel, finagle, varnish, ha-proxy, func, capistrano, puppet, jenkins

Quora

aws ec2, s3, ubuntu, cloudfront, python, pylons, paste, tornado, thrift, comet server, memcached, haproxy, nginx

Pinterest

python, django, tornado, node.js, rabbitmq, nginx, haproxy, varnish, memcached, membase, redis, my-sql, mrjob, hadoop(elastic map reduce)

Instagram (Korean)

aws, s3, ebs, cloudfront, ubuntu, django(high-cpu extra-large), gunicorn, fabric, gearman, pyapns, twisted, postgre-sql(quadruple extra-large), mdadm(sofeware raid with ebs), repmgr, pgbouncer, redis, memcached, node2dm, munin, pingdom, pagerduty, sentry

Pulse

aws, ec2, s3, elb, tornado, scribe, mrjob, node-readability, haproxy, tornado, gae, mapreduce, django(appengine), google cloud storage, memcache, redis

Netflix

aws(ec2, s3, ebs, rds, dynamodb, sdb, sqs, sns, emr, elb, eip, vpc, direct-connect, iam), java, tomcat, mongodb, my-sql, casandra, hadoop, zookeeper, evcache, asgard, groovy, grails, zuul, priam and more netflix opensouces)

Spoqa

linux 2.6, nginx, uwsgi, s3, dotcloud, mysql, redis, celery

StyleShare

ubuntu 12.04, aws, ec2, s3, elb, nginx, werkzeug, flask, postgre-sql, pgpool, memcached, gevent, celery, rabbitmq, fabric, boto, exceptional, flask-exceptional

Gae9(개9)

rabbitmq, celery, phash

TheBorn Coupon(더본쿠폰)

ubuntu(12.04), nhn ncloud, django, apache, mod_wsgi, ms-sql, memcached, fabric, south, wand, rsync, py-bcrypt, python-gcm, apns

REFERENCES

INFRA, PLATFORMS, FRAMEWORKS

DATABASE, STORAGE

  • PostgreSQL: most advanced open source database.
  • repmgr: open source tools that helps DBAs and System administrators manage a cluster of PostgreSQL databases.
  • pgpool Wiki: middleware that works between PostgreSQL servers and a PostgreSQL database client.
  • PgBouncer: lightweight connection pooler for PostgreSQL.
  • SQLAlchemy: Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL.
  • South: intelligent schema and data migrations for ​Django projects.
  • twitter/gizzard · GitHub: flexible sharding framework for creating eventually-consistent distributed datastores
  • cassandra: used for high velocity writes, and lower velocity reads
  • hadoop: process unstructured and large datasets, hundreds of billions of rows.
  • vertica: used for analytics and large aggregations and joins so they don't have to write MapReduce jobs. (twitter)
  • mrjob: Run MapReduce jobs on Hadoop or Amazon Web Services
  • Apache Solr: popular, blazing fast open source enterprise search platform from the Apache LuceneTM project
  • fatcache: Memcache on SSD.
  • google cloud storage: Store, access and manage your data on Google’s storage infrastructure. Take advantage of the scale and efficiency we have built over the years.
  • haystack
  • Netflix/EVCache

DEPLOY, MONITORING, UTILITIES

  • Fabric: library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.
  • boto/boto · GitHub: python interface to aws
  • capistrano/capistrano · GitHub: Remote multi-server automation tool.
  • puppetlabs/puppet · GitHub: Server automation framework and application.
  • Exceptional: Exceptional tracks errors in web apps. It reports them in real-time.and gathers the info you need to fix them fast.
  • jzempel/flask-exceptional · GitHub: Exceptional extension for Flask.
  • rsync: utility that provides fast incremental file transfer.
  • Nagios: The Industry Standard in IT Infrastructure Monitoring
  • Munin: networked resource monitoring tool that can help analyze resource trends
  • AWStats: powerful and featureful tool that generates advanced web, streaming, ftp or mail server statistics, graphically
  • pingdom: Website monitoring. Monitor your server and network uptime and performance for free.
  • sentry: realtime error logging and aggregation platform
  • pagerduty: SaaS IT on-call schedule management, alerting and incident tracking.
  • scribe: server for aggregating log data that's streamed in realtime from clients. It is designed to be scalable and reliable
  • Sphinx: tool that makes it easy to create intelligent and beautiful
  • pHash.org: perceptual hash library
  • dahlia/wand · GitHub: The ctypes-based simple ImageMagick binding for Python.
  • py-bcrypt: strong password hashing for Python.
  • mdadm: manage MD devices aka Linux Software RAID.
  • twitter/snowflake · GitHub: network service for generating unique ID numbers at high scale with some simple guarantees
  • node-readability: Server side readability with node.js
  • Netflix/asgard
  • Netflix/Priam
  • Netflix/zuul
Add a custom footer
Add a custom sidebar

Clone this wiki locally