Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Wonderful reusable code from Twitter
Scala Java Python Other

csl: Publish externally with pants, sbt, and friends

Problem

It's difficult to publish util, finagle, etc. because we haven't
updated them for the new maven-less world.

Solution

Scripts that make it easy to publish with sbt and pants.  Bumped
the server-set version because we stopped publishing thrift
externally in ci.  Changed an ExitGuard test which broke under
weird sbt conditions.

Result

We're in a position where we can start publishing finagle, etc
again!

RB_ID=633570
latest commit 0c4fabfeb9
@mosesn mosesn authored jenkins committed
Failed to load latest commit information.
codegen [util-test] Easy argument capture for Mockito mocks.
project csl: Publish externally with pants, sbt, and friends
util-app util-app: quote flags in a shell-compatible way
util-benchmark util-core: Add `Once` for doing something exactly once
util-cache util: Cleanup various compile time warnings
util-class-preloader [split] Remove more poms
util-codec birdcage: Split out 3rdparty deps to be opened
util-collection util-collection: Add copy to RecordSchema.Record.
util-core csl: Publish externally with pants, sbt, and friends
util-eval Added ability to override `Reporter` used during compilation. See issue
util-events [flight-recorder] Download Sink in Trace Event format
util-function Add helpers for Java 8 lambda integration
util-hashing birdcage: Split out 3rdparty deps to be opened
util-jvm util: Cleanup various compile time warnings
util-logging [trivial] Event sink default should match root logger level
util-reflect birdcage: Split out 3rdparty deps to be opened
util-registry finagle-core: Removes old registry entries when they're no longer ref…
util-stats util: Cleanup various compile time warnings
util-test Add util-test/BUILD, enabling pants testing of util-test.
util-thrift util/finagle: Small changes to avoid allocations
util-zk-common birdcage: Split out 3rdparty deps to be opened
util-zk-test util-zk-test: Thread maxClients through to underlying Factory
util-zk kill ver
util/util-thrift/src/test/scala/com/twitter/util util-thrift: add unit tests for {Binary,Compact}ThriftCodec
.gitignore add sbt-launch.jar to gitignores
.travis.yml Fix problems with SBT and Travis CI builds, update Scala and SBT vers…
CHANGES csl: Publish externally with pants, sbt, and friends
CONFIG.ini [split] s/GUILD/CSL/ for all CONFIG.ini's
CONTRIBUTING.md Update documentation about contributing
GROUPS [split] Use new git-review with simplified OWNERS/GROUPS
LICENSE [split] Fixes race in channel.send(); fixes the meaning of filter(); …
OWNERS util, twitter-server: Add koliver to OWNERS
README.markdown Update documentation about contributing
sbt Fix problems with SBT and Travis CI builds, update Scala and SBT vers…
updatedocs.bash [split] util-doc: Fix updatedocs.bash to update new util docs

README.markdown

A bunch of idiomatic, small General Purpose tools.

Build Status

See the Scaladoc here

Using in your Project

Pre-compiled jars for each set of tools (util-core, util-collection etc) are available in the Twitter Maven repository, here: http://maven.twttr.com/

We use Semantic Versioning for published artifacts.

An example SBT dependency string for the util-collection tools would look like this:

val collUtils = "com.twitter" %% "util-collection" % "6.23.0"

Units

Time

import com.twitter.conversions.time._

val duration1 = 1.second
val duration2 = 2.minutes
duration1.inMillis // => 1000L

Space

import com.twitter.conversions.storage._
val amount = 8.megabytes
amount.inBytes // => 8388608L
amount.inKilobytes // => 8192L

Futures

A Non-actor re-implementation of Scala Futures.

import com.twitter.util.{Future, Promise}

val f = new Promise[Int]
val g = f map { result => result + 1 }
f.setValue(1)
g.get(1.second) // => This blocks for the futures result (and eventually returns 2)

// Another option:
g respond { result =>
  println(result) // => prints "2"
}

// Using for expressions:
val xFuture = Future(1)
val yFuture = Future(2)

for (
  x <- xFuture
  y <- yFuture
) {
  println(x + y) // => prints "3"
}

Collections

LruMap

The LruMap is an LRU with a maximum size passed in. If the map is full it expires items in FIFO order. Reading a value will move an item to the top of the stack.

import com.twitter.util.LruMap

val f = new LruMap[Key, Value](15) // this is of type mutable.Map[Key, Value]

Google MapMaker

import com.twitter.util.MapMaker

val map = MapMaker[Key, Value] { config =>
  config.weakKeys()
  config.weakValues()
} // this is of type mutable.Map[Key, Value]

Object Pool

The pool order is FIFO

A pool of constants

val queue = new mutable.Queue[Int] ++ List(1, 2, 3)
val pool = new SimplePool(queue)
// Note that the pool returns Futures, it doesn't block on exhaustion.
pool.reserve()() mustEqual 1
pool.reserve { item =>
  println(item) // prints "2"
}

A pool of dynamically created objects

Here is a pool of even-number generators. It stores 4 numbers at a time:

val pool = new FactoryPool[Int](4) {
  var count = 0
  def makeItem() = { count += 1; Future(count) }
  def isHealthy(i: Int) = i % 2 == 0
}

It checks the health when you successfully reserve an object (i.e., when the Future yields).

Eval

Dynamically evaluates Scala strings and files.

This is motivated by the desire to have a type-safe alternative to textual configuration formats such as YAML, JSON, or .properties files. Its advantages over these text formats are

  • Strong typing and compiler checking. If it doesn't compile and doesn't conform to the type you expect, you get an exception
  • The full power of Scala in your config. You don't have to use it. But you can.

in config/Development.scala

import com.xxx.MyConfig

new MyConfig {
  val myValue = 1
  val myTime = 2.seconds
  val myStorage = 14.kilobytes
}

in Main.scala

import com.xxx.MyConfig

val config = Eval[MyConfig](new File("config/Development.scala"))

Version 6.x

Major version 6 introduced some breaking changes:

  • Futures are no longer Cancellable; cancellation is replaced with a simpler interrupt mechanism.
  • Time and duration implement true sentinels (similar to infinities in doubles). Time.now uses system time instead of nanotime + offset.
  • The (dangerous) implicit conversion from a Duration to a Long was removed.
  • Trys and Futures no longer handle fatal exceptions: these are propagated to the dispatching thread.

Future interrupts

Method raise on Future (def raise(cause: Throwable)) raises the interrupt described by cause to the producer of this Future. Interrupt handlers are installed on a Promise using setInterruptHandler, which takes a partial function:

val p = new Promise[T]
p.setInterruptHandler {
  case exc: MyException =>
    // deal with interrupt..
}

Interrupts differ in semantics from cancellation in important ways: there can only be one interrupt handler per promise, and interrupts are only delivered if the promise is not yet complete.

Time and Duration

Like arithmetic on doubles, Time and Duration arithmetic is now free of overflows. Instead, they overflow to Top and Bottom values, which are analogous to positive and negative infinity.

Since the resolution of Time.now has been reduced (and is also more expensive due to its use of system time), a new Stopwatch API has been introduced in order to calculate durations of time.

It's used simply:

val elapsed: () => Duration = Stopwatch.start()

which is read by applying elapsed:

val duration: Duration = elapsed()

Contributing

The master branch of this repository contains the latest stable release of Util, and weekly snapshots are published to the develop branch. In general pull requests should be submitted against develop. See CONTRIBUTING.md for more details about how to contribute.

Something went wrong with that request. Please try again.