Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Work in progress!!! Apologies Please do note, there is more coming...this is just the inception of this library in order to allow java code sharing between internal projects and the man open source ones we work on (hive, hdfs, mapreduce, hbase, zookeeper, giraph, and more coming)
This is just a smattering of the java code inside facebook. Some major projects such as Hive have made it it in the past, largely because they were not intertwined with other components that either were not to be open-sourced, too-hard to abstract out, or if you made it that far, a reference implementation would be a major project.
A few notable Java projects have been created with the intent to open source:
- Calligraphus (aka "java scribe")
Table of Contents
PENDING: wiki will come up with guidelines see the jcommon development page for info on checking out and committing.
You can see the coding standards page for now
Bear with us, some of you will see some components and point out it exists in Guava, or your own library you open sourced. We actively look for code out there to solve the current problem, so we could get to the next one, usually bigger. That said, much of this was written when Guava was till on r04, and many bugs in compute map as well as some performance issues kept us away. In other cases, we probably just didn't know about your library.
This is its own module mostly for circular dependency resolution, but also has the theme of static methods operating on Iterators, Lists, and Arrays. Some interfaces are defined related to those and to translate from our Mapper to Guava's Function (irony, anyone?)
This contains libraries around, well, collections. Varies from highly concurrent data structures optimized for performance and parallel processing, to some interfaces that allow data structures to be backed by primitive arrays  in order to save memory (see SimpleHeap or Array and reference implementations).
- CounterMap: convenient for thread-safe counters on any object that properly implements hashcode()/equals() (ie works in a hashmap). This has the boiler-plate
- SetMap : similar, except the data are Sets for each key. Using Generics to capture the set type, not just X, S extends Set
- ConcurrentCache was particularly useful, even with Guava's compute-map up through version 11 we couldn't use it as it would not do expiration so long as you only called get(key). This was fixed in r11, but we've evolved away from a pure "Map". Now I hear version 13 has something similar to this (haven't had time to check). We have the "Core" one that is optimized for concurrency and memory, while the time-expiring one has not yet been memory-gutted.
- ByteArray is our take on getting arrays into data structures requiring hashcode()/equals(), but not storing >= 20 bytes of overhead for a HeapByteBuffer.
Primarily introduces a "Datum" interface which says the Datum object will tell you how to handle it, mostly, but let you abuse it if you want. The idea is to store the 'natural' type for a type. Take LongDatum. It stores a 'long', so is efficient for fetching for math ops, but is also convertible to a String or float/double if needed.
Once we open source Puma, you'll see how much better this is than passing around Object or "Object..." and returning Object. You can ask the Datum about its type (what it is, is it whole-number, etc). This clears out the classic instanceof checks and leads to more readable code, and potential efficiencies if you see NumberCachingDatum.
More complex abstractions lay in the fact a "Datum" could be a List or a Map. If you coded perl, this model is inspired by it (love it or hate it). It also maps to JSON nicely and we plan to release our code to do JSON serde work here.
The crux of this library is to minimize the # of threads created in the JVM. They are, in fact, costly. Too many threads at once, or lots of creation/destruction adds overhead. This is a so-far unique set of classes that start to treat some # of threads (say, the # of core) like cpus and allows sharing of those. However, it's implemented with the java ExecutorService framework, so anything that works with that can use these "virtual executors" and "virtual threads". This was one of the huge wins for Calligraphus over scribe and why it was more efficient in that it pooled threads more efficiently.
Lots of useful classes around Executors. They allow you to use a single cached thread pool and have "virtual' Executors (basically just another Queue, but with the same semantics as a real Executor as far as shutdown(), awaitTermination(), etc. The most interesting classes:
- ExecutorServiceFront and ExecutorServiceFrontBuilder (former known as "ESF")
- does time-slice sharing in the case you have fewer total allowed threads in a base pool, but sum of layers on top add up to more. Example, 12 cpu threads in a cpu pool (which is a front on top of a cached thread pool with a limit). If you then have two executors on top of this one, that each say, ask for 10, then make sure the 12-max one uses time-sharing. Depending on your task lengths (usually prefer short--manage long-running cron-like tasks differently), a slice of even 1s is fine. If you want finer granularity, ty 100-500ms and vary. We use in puma havily
- several classes use this as the core to simulate an executor. While the ESF provides just the queue and drainers ("virtual threads"), it needs to be combined with UnstoppableExecutorService to make it useful. This descriptively named class simply means if you wrap an ExecutorService in it, the shutdown() calls will have the same semantics, but the underlying Executor can't be stopped. It's useful to wrap an Executor you want to share with a foreign libraries code to avoid premature shutdown. This class is the core of UnstoppableExecutorService and UnstoppableScheduledExecutorService
- real use cases: In other systems such as puma, we use one root 'cachedThreadPool' and overlay several UnstoppableExecutorServiceFront instances with some # of max threads each. We use this to enforce resource constraints: the one for cpus has a reasonable value, one for IO against resources such as MySQL or HBase has one, ones for executing API queries, and so on. You get thread-caching with a small penalty in the queue; but this is the same as a regular executorService, but you get the thread re-use.
- Same as collections: Many of these were added before we started using Guava, also. Caution: sometimes we found guava's impl of similar data structures either less memory efficient or slower. Prefer Guava to writing your own, but when it comes to existing ones here, compare perf/memory and choose the right one.
- update (2012-06-01): Vanilla Java (aka Huge Collections) doesn't solve the memory efficiency issue, but does propose interesting ideas of memory mapping the data, and letting you define your data structure as an interface. We still plan to implement structs on native memory in Java 7, before mechanisms in Java 8 will effectively allow them.
Our libraries for parsing and managing JSON config files. Mostly convenience, not aware of anything off the top of my head that does this, but I'm sure every company has their own if they use JSON. Uses some reflection to allow user-defined "beans", but the jackson parsing libraries are far more advanced and should probably be preferred. Our ConfigProvider and ConfigAccessor have just been nice adapters over whatever JSON parser used.
Only one class really here but it's critical in java apps—a clean shutdown means every thread gracefully terminates. This is a class now handling that by allowing an extensible number of "stages" to be registered and objects indicate the stage they want to be shutdown in. Example that drove this: flushing of data from memory had to happen before closing streams to HDFS (may sound simple, but there's a global close that is done at the end to close a FileSystem, not a file). Puma has benefitted from it as well by making sure anything that uses executors happens before we shut them down. Calligraphus could make sure a router server sharing a jvm with a writer shutdown first.
Facades over log4j that improve performance and add convenience as it defers expensive sprintf() format strings. It does allow the old style so as not to get unintended log statements (think about the varargs form matching the old "log.info(msg, exception)"). It also includes a novel (I haven't seen yet) "time sampling logger" which solves the problem of log spew by saying "log at most 1 line per 30 seconds".
This is sparse, bootstrapping of doing native memory manipulation such as slab allocators to do "structs" and avoid 1. java GC 2. java object memory overhead. Only has a way to get the Unsafe class. Danger! but efficiency. Anecdotal: using Unsafe.putLong() vs a DirectByteBuffer.putLong(), the former is an order of magnitude faster in the cases we've measured. The latter does more checking that results in more than the single store instruction a putLong maps to on a 64-bit system.
Originally written for the scribe replacement, Calliigraphus, as there were no Java stats counters in existence at Facebook in August of 2010. This started with the rates/sum being added by srash from DataFreeway, then Charles Thayer from ODS added Gauge, and I believe Eric Huang added Min/Max and he or someone else put those three together into the useful "spread" counter.
Other utils are present, and we've recently added histogram and percentile. Variations on a bounded topk exist. We have a topk approximation that we've yet to pull in.
The stats here are pretty standard: count, rate, min/max, average, percentile/histograms, variations of streaming top-k or max-k. However, we do many (esp histogram/percentile) in a VERY efficient way to allow 2M+ calls per second and works with a unique data structure that gives you arbitrary percentile/histograms (you don’t' have to pre-specify). Simpler stats like count/rate/sum/min/max/average are 80M+/second.
testing helpers. These vary from useful Mock objects to test concurrent race conditions (MockExecutor). A couple other tests are related and for doing things in other threads during tests in a concise manner
This is a catch–all for other classes. Includes a useful TimeInterval type that we found helpful. It was helpful in working with JodaTime (excellent 3rd party lib we use) as there are two notions of some span of time: one that is a fixed length, and one that varies based on start (think DST—23 hour day, 25 hour day). There are also some other utility objects that ostensibly should be in concurrency, but landed here
testing.... This has some well-known classes everyone has hard of such as a leader election. However, due to a lack of using interfaces in the zookeeper code base, nor easy to extend code (violations of the open-closed principle; and mockito would have been a pain), we basically put a facade interface over Zookeeper. We wanted all of our code testable at the unit level, as well as component and end-to-end/integration.
testable code leads to other good stuff This has a nice side-effect of "RecoverableZoookeeper" being a popular class used in FB's HBaase. It takes the idea that if you send an idempotent command to zookeeper and want it to keep trying until it succeeds, this will do so. There are boundary/edge cases for non-idempotent ops: those that involve sequences. Here, we took the same apporach unix took to deadlock at the OS level: ostrich (nothing, I see nothing). Basically, your operation is idempotent only to the point that you will eventually create a sequence node (in the ephemeral; the non-ephemeral sequence, we push back up to the application as it's not known if it was created)
One of the great abstractions to come out of this, though, was ZkApplication.
ZkApplication is an abstract base class that provides a template-and-hook style ZooKeeper state management framework.
- Automatic connection and reconnection to ZooKeeper on disconnects and expirations.
- Issues initialize(), repair(), and expire() callbacks to the application as various connection events occur.
- Models application state with a finite state machine. States are queryable from subclasses.
<u>PRESTART</u> This is the application's initial state before the user issues a start() command to begin the application <u>DISCONNECTED</u> Application is not connected to ZooKeeper and does not have any application state set in ZooKeeper (e.g. watches or ephemeral nodes). Entry to this state automatically triggers a connection loop that retries until successful. <u>CONNECTED</u> Application is connected to ZooKeeper, but application state in ZooKeeper has not been fully initialized. Entry to this state automatically triggers callbacks to the initialize() abstract method, looping until initialize() returns true. <u>FUNCTIONAL</u> Application is connected to ZooKeeper and has full application state initialized in ZooKeeper. At this point the application should be fully functioning. <u>SAFEMODE</u> Application was successfully initialized, but became disconnected. Since many applications will cache ZooKeeper state, this state signifies a cache read-only mode where ZooKeeper is unavailable. Entry to this state will automatically trigger a connection loop that retries until reconnected, or until the session expires. <u>SAFEMODE_REPAIR</u> Application in safemode was successfully reconnected to ZooKeeper without expiration. Entry to this state will automatically trigger callbacks to the repair() abstract method, looping until repair() returns true. While initialize() and repair() may do the same things, we make this distinction as it is often possible to optimize the repair method to only repeat failed commands. <u>SHUTDOWN</u>
* Application has been shut down by the user via the shutdown() method. * * ======================= STATE TRANSITION DIAGRAM ===================== * * The state transition diagram appears as follows: * * PRESTART (expire*) * | (start) | * v | * DISCONNECTED <---- --> SAFEMODE * | (connect) / | (connect) * v / v * CONNECTED (dc'ed) / SAFEMODE_REPAIR * \ / / * \ / / * (init) \ / / (repair) * \ / / * v / / * FUNCTIONAL <----- * * * (shutdown*) -----> SHUTDOWN * * NOTE: '*' denotes an event that unconditionally leads to a specific state, * regardles of the pre-existing state.
including test coverage reports: PENDING