Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Commits on Apr 22, 2015
  1. @mosesn

    csl: Publish externally with pants, sbt, and friends

    mosesn authored jenkins committed
    Problem
    
    It's difficult to publish util, finagle, etc. because we haven't
    updated them for the new maven-less world.
    
    Solution
    
    Scripts that make it easy to publish with sbt and pants.  Bumped
    the server-set version because we stopped publishing thrift
    externally in ci.  Changed an ExitGuard test which broke under
    weird sbt conditions.
    
    Result
    
    We're in a position where we can start publishing finagle, etc
    again!
    
    RB_ID=633570
  2. @luciferous

    [trivial] Event sink default should match root logger level

    luciferous authored jenkins committed
    Problem
    
    EventSink defaults to capturing the root logger at Level.ALL.  This is
    problematic because some log messages are computed lazily, and it would
    be too easy to unwittingly force these messages with EventSink's
    defaults.
    
    Solution
    
    Match the log level of the root logger. This is what service owners
    would set to a reasonable default.
    
    RB_ID=639266
  3. @mosesn

    finagle-exp: Make BackupRequestFilter allocate less

    mosesn authored jenkins committed
    Problem
    
    BackupRequestFilter was allocating right, left, and center.
    
    Solution
    
    Remove as many allocations as possible while maintaining correctness
    and clarity.
    
    RB_ID=634734
Commits on Apr 20, 2015
  1. @kevinoliver

    util-core: Add Java to immutable Scala collection conversions

    kevinoliver authored jenkins committed
    Problem
    
    It is difficult to convert from a `java.util.Map` to a
    `scala.collection.immutable.Map` from Java due to the implicit
    evidence needed by `toMap`.
    
    Solution
    
    Introduce `c.t.u.javainterop.Scala` methods to help Java devs convert
    into immutable Scala collections. While the other variants are not
    strictly necessary from Java, the uniform API should make it easier
    for devs.
    
    RB_ID=634464
  2. @travisbrown

    Update SBT's ScalaCheck and ScalaTest versions to match 3rdparty

    travisbrown authored jenkins committed
    RB_ID=627742
  3. @mosesn

    finagle-core: Removes old registry entries when they're no longer ref…

    mosesn authored jenkins committed
    …erenced
    
    Problem
    
    Although creating many finagle clients is typically an antipattern,
    it's occasionally necessary, for example for web crawlers or link
    shorteners.
    
    Solution
    
    We remove old registry entries when they're no longer referenced.
    This requires adding a \`remove\` method to Registry, and hooking
    it up to closing the ServiceFactory you're handed.
    
    Result
    
    As long as users close their resources, registry "just works".
    
    RB_ID=625826
Commits on Apr 13, 2015
  1. @kevinoliver

    util: Cleanup various compile time warnings

    kevinoliver authored jenkins committed
    Problem
    
    There are a lot of compiler warnings in util.
    
    Solution
    
    Fix (most of) them.
    
    The vast majority that remain are own deprecated code which we should
    fix when the deprecated code is removed.
    
    RB_ID=632242
  2. @luciferous

    [util-core] AsyncStream drop(0) should return this

    luciferous authored jenkins committed
    Problem
    
    In AsyncStream, drop always returns an empty stream for drop(0).
    
    Solution
    
    Return "this" for drop(0).
    
    RB_ID=632262
  3. @kevinoliver

    util/finagle: Small changes to avoid allocations

    kevinoliver authored jenkins committed
    Problem
    
    In various places we use methods that rely on Scala's implicits to
    enrich them. This typically at the cost of an unnecessary object
    allocation. There are some other simple changes to be made that
    avoid allocations as well.
    
    Solution
    
    Use methods that do not require allocations. Some examples:
    
    * String.nonEmpty => !String.isEmpty
    * String.size => String.length
    * StringBuilder.size => StringBuilder.length
    * Array.size => Array.length
    * Seq.empty => Nil
    
    RB_ID=630964
  4. @vkostyukov

    util-core: Improve Java-friendliness of Activities

    vkostyukov authored jenkins committed
    RB_ID=630184
  5. @bmdhacks

    util-core: add Time.fromFractionalSeconds(Double)

    bmdhacks authored jenkins committed
    Problem
    
    Javascript doesn't have integer types, so there's no real ideal way of
    encoding a precise timestamp in a single field.  Strings are an option,
    but another common method is to use a Double precision epoch time in
    seconds.  This provides enough accuracy for measuring most things we
    care about, but then it's hard to deal with in scala.
    
    Solution
    
    Add Time.fromFractionalSeconds(Double) to parse json floating-point
    timestamps.
    
    RB_ID=630188
  6. @luciferous

    Introduce AsyncStream

    luciferous authored jenkins committed
    AsyncStream (like Spool) is a lazy list with its tail inside a Future.
    But unlike Spool it has some properties that make it easier to use.
    
    * Composable with Future, Option, and Seq via "lift" functions
      fromFuture, fromOption and fromSeq.
    * Associative flatMap -- the evaluation order of flatMap doesn't affect
      the constructed stream.
    * Associative side effects -- the evaluation order of flatMap doesn't
      affect the order of side effects when the stream is evaluated.
    
    Example usage:
    
    Given a service call `getUser(id: UserId): Future[User]` and a list of
    `ids: Seq[UserId]`. This is an AsyncStream of users that are not
    rate-limited, given a Seq of user IDs.
    
        def users(ids: Seq[UserId]): AsyncStream[User] = for {
          id <- fromSeq(ids)
          limited <- run(checkLimit(id))
          if !limited
          user <- run(getUser(id))
        } yield user
    
        users(123L +: Nil).foreach(u => println(u.name))
    
    The equivalent Spool might be something like:
    
        def checkLimitAndId(id: UserId): Future[(UserId, Boolean)] =
          checkLimit(id).map(limited => id -> limited)
    
        def users(ids: Seq[UserId]): Future[Spool[User]] =
          ids.toSpool.mapFuture(checkLimitAndId).flatMap { idLimits =>
            idLimits.filter { case (_, limit) => limit }.flatMap { filtered =>
              filtered.flatMap { case (id, _) =>
                getUser(id).map(u => Seq(u).toSpool)
              }
            }
          }
    
        users(123L +: Nil).onSuccess { spool =>
          spool.foreach(u => println(u.name))
        }
    
    Smokestack
    
    identity = fromSeq andThen toSeq andThen Await.result(_)
    
    identity        1000000 3768 bytes/iter
    map             1000000 5832 bytes/iter
    flatMap         1000000 7688 bytes/iter
    filter          1000000 4032 bytes/iter
    takeWhile       1000000 3832 bytes/iter
    spool.identity  1000000 3880 bytes/iter
    spool.map       1000000 6728 bytes/iter
    spool.flatMap   1000000 12096 bytes/iter
    
    RB_ID=594106
Commits on Apr 6, 2015
  1. @travisbrown

    Use sbt-unidoc plugin and remove out-of-date copy-pasted version

    travisbrown authored jenkins committed
    Problem
    
    A couple of months ago I set up our SBT builds to include API doc links in the
    POMs, which makes it possible for e.g. Finagle's API docs to link to Util's.
    This didn't actually work because we were using an old copy-pasted version of
    Unidoc, and unfortunately the same bug was present in the sbt-unidoc plugin.
    
    Solution
    
    My fix for the sbt-unidoc bug is now available in the 0.3.2 release, so we can
    switch to using it. I've also added an API doc link to TwitterServer's SBT
    config, even though historically we haven't published TwitterServer's API docs
    (I'd like to start).
    
    Result
    
    We can build Finagle's API docs with Unidoc and have references to classes from
    Util and the Scala standard library automatically linked to their definitions in
    those projects' API docs.
    
    RB_ID=627082
  2. @stuhood

    birdcage: Remove empty util-cache/java target

    stuhood authored jenkins committed
    java_library targets without sources are basically target aliases (`target(name='..', ..)`), which don't have provides clauses.
    
    I suspect that you meant to include some sources in this target? But if you didn't it shouldn't be used, because it would create false dependencies between the libraries that it depends on.
    
    RB_ID=626970
    TBR=true
  3. @travisbrown

    Fix problems with SBT and Travis CI builds, update Scala and SBT vers…

    travisbrown authored jenkins committed
    …ions
    
    Problem
    
    SBT builds for Finagle and associated projects were failing for a number of
    reasons, including third-party dependency versions that had been updated for
    Pants but not SBT, new tests that are run differently by the SBT test runner,
    and changes to JSON serialization that don't work the same on Scala 2.10 and
    2.11. Our SBT and Scala versions were also out of date.
    
    Also Travis CI's container-based infrastructure currently has a bug that causes
    the build to crash when getting the local hostname with IPv4 on OpenJDK 7
    (reported in #3120 on GitHub). Moving back to the standard Travis infrastructure
    is a trivial change, so I've included it here.
    
    Solution
    
    I've updated the SBT and Scala versions throughout, and have updated third-party
    dependency versions to match the configuration in 3rdparty (note that I've
    followed the last update to scrooge/project/build.properties in not updating the
    custom SBT script for Scrooge). TwitterServer now uses scala.xml, which is a
    separate module in Scala 2.11, so I've added a conditional dependency in the SBT
    build.
    
    The new ExitGuard tests were failing on SBT because clients were being left open
    in other test suites. I've daemonized all of these clients so that they don't
    create exit guards.
    
    In JsonSinkTest in TwitterServer I've desugared an implicit conversion
    provided by a view bound because Scala 2.11 prohibits Object as the target of an
    implicit conversion.
    
    Because Jackson doesn't correctly serialize Scala case classes that are defined
    locally in vals on Scala 2.11, I've moved three case classes out of vals and
    into the enclosing objects, changing names where necessary to avoid collisions,
    and making each case class private to the object.
    
    I've also switched all projects with a custom SBT script back to the standard
    Travis CI infrastructure.
    
    Result
    
    We're on the most recent versions of Scala and SBT, the SBT builds actually
    work, and Travis CI won't fail spuriously on OpenJDK 7.
    
    RB_ID=619988
  4. @kevinoliver

    Improve Java APIs in a variety of areas.

    kevinoliver authored jenkins committed
    RB_ID=624900
  5. @kevinoliver

    util-stats: Fix deadlock in CumulativeGauge

    kevinoliver authored jenkins committed
    Problem
    
    Lock acquisition order was not always consistent between
    `CumulativeGauge` and `StatsReceiverWithCumulativeGauges` which can
    lead to deadlocks.
    
    Solution
    
    Remove synchronization from `StatsReceiverWithCumulativeGauges`.
    
    RB_ID=624714
  6. @travisbrown

    Update documentation about contributing

    travisbrown authored jenkins committed
    Problem
    
    The contributing documentation currently recommends submitting pull requests
    against master.
    
    Solution
    
    Update the contributing documentation to reflect the current workflow, and add
    a note about branches to the main README.
    
    Signed-off-by: Travis Brown <tbrown@twitter.com>
    
    RB_ID=623652
  7. @dschobel

    birdcage/*: migrate to finagle-kestrelx

    dschobel authored jenkins committed
    Migrate birdcage from finagle-kestrel to finagle-kestrelx as part
    of Twitter's netty4 migration.
    
    RB_ID=618432
  8. @roanta

    finagle-benchmark: convert thrift benchmarks to jmh / smokestack

    roanta authored jenkins committed
    RB_ID=623478
  9. @mariusae

    finagle-mux: rewrite server dispatcher

    mariusae authored jenkins committed
    Problem
    
    My previous change to add metrics made it very obvious that
    the Mux server dispatcher was becoming increasingly difficult
    to reason about, especially in terms of session state.
    
    Solution
    
    Rewrite the server dispatcher in a more sustainable way. Indeed
    this process found more races. Among the changes are:
    
     *	Separate out request processing from session management
     *	Remove unnecessary configuration (canDispatch, ping), and
    	move their functionality into a filter chain.
     *	Introduce a separate transaction tracker that manages draining.
     *	Introduce a fine-grained session state so that we can tell the
    	difference between draining, client, and server hangups.
    
    This also introduces 4 new metrics to track draining stats:
    
    **draining**
      the number of times the server has initiated session draining
    
    **drained**
      the number of times the server has succesfully completed the draining protocol within its allotted time
    
    **clienthangup**
      the number of times sessions have been abruptly terminated by the client
    
    **serverhangup**
      the number of times sessions have been abruptly terminated by the server
    
    Result
    
    Much easier to reason about the server dispatcher code and session states.
    
    Smokestack
    
    old:
    	mux	1000000	16555 bytes/iter
    	muxsess	1000000	30348 bytes/iter
    
    new:
    	mux	1000000	16867 bytes/iter
    	muxsess	1000000	31555 bytes/iter
    
    RB_ID=580754
  10. @luciferous

    [flight-recorder] Download Sink in Trace Event format

    luciferous authored jenkins committed
    Problem
    
    Currently the event capture can be displayed as an HTML table or as raw
    JSON. We want to better understand the types of events that will be
    collected in order to design useful presentations of those events. In
    the interim, however, we have no solution for users who want a more
    sophisticated presentation.
    
    Solution
    
    Trace Viewer[1] is built into Chrome, and greatly improves upon the
    table presentation. We provide an endpoint for downloading an event
    capture in Trace Event[2] format, which can then be loaded into Trace
    Viewer.
    
    [1] https://github.com/google/trace-viewer
    [2] https://docs.google.com/a/twitter.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/edit#heading=h.yr4qxyxotyw
    
    RB_ID=618082
  11. @kevinoliver

    Problem

    kevinoliver authored jenkins committed
    Scala creates Functions instead of a PartialFunction when used
    anonymously as in:
    
      Future.onFailure { case NonFatal(e) => ... }
    
    This meant that the special handling introduced in RB_ID=588992 failed
    to actually help. The unit test "lied" because it passed explicit
    PartialFunctions instead of the more idiomatic anonymous ones.
    
    Solution
    
    Revert the changes introduced in RB_ID=588992 and fix the call sites
    that were using it incorrectly.
    
    RB_ID=620938
Commits on Mar 30, 2015
  1. @mariusae

    finagle-core: eliminate some allocations in context processing

    mariusae authored jenkins committed
    Problem
    
    The deadline branch introduced a standard context that is now shipped
    in basically all clients and servers. This resulted in a relatively steep
    allocation regression in the mux smoke tests (~20%). With this change,
    we recoup all of it.
    
    Solution
    
    Reduce allocation during key processing and comparison:
    
    1. Make sure context keys are ByteBuffers so that we can;
    2. compare them directly without copying them.
    
    On master, we get this in the Mux allocation test
    
    	before 8552 bytes/iter
    	after 18486 bytes/iter
    
    The effect is (much) more pronounced in the deadlines branch.
    
    RB_ID=606326
    TBR=true
Commits on Mar 23, 2015
  1. @travisbrown

    Change class-to-trait inheritance to self type

    travisbrown authored jenkins committed
    Problem
    
    The behavior of traits that extend classes in Scala seems to be underspecified
    and buggy, and this code breaks between 2.11.4 and 2.11.5 (see my new SI-9239).
    
    Solution
    
    Switching to a self type works on 2.11.5, isn't flagged by MiMa as an ABI
    change, and can't break code outside of Util because the trait is sealed.
    
    Result
    
    We can upgrade our 2.11 build to 2.11.6.
    
    RB_ID=613760
  2. util-zk-test: Thread maxClients through to underlying Factory

    Chris Chen authored jenkins committed
    Problem:
    maxClients was added as a argument but was not threaded through to
    the underlying Factory.
    
    Solution:
    Thread through maxClients to underlying factory.
    
    Result:
    maxClients is now set on the underlying factory.
    
    RB_ID=613476
  3. @luciferous

    [util] Use proxyTo instead of become

    luciferous authored jenkins committed
    Problem
    
    In the case where it is passed a promise, Promise.become merges the
    listeners of two promises, an operation which requires modifying the
    internal state of both Promises. The complication arises when the
    promises each have an interrupt handler: one gets overwritten.
    
    Consider the following, which demonstrates the behavior of `become` on
    promises with interrupt handlers.
    
        val a, b = new Promise[Unit]
        a.setInterruptHandler { case _ => println("A") }
        b.become(a)
        b.setInterruptHandler { case _ => println("B") }
        a.raise(new Exception)
    
    This prints "B", the action in the interrupt handler for `b`, which is
    unexpected because we raised on `a`.
    
    Solution
    
    A cleaner solution is to use proxyTo, which does not merge listeners,
    but relies only on the standard Future.respond. We can set an interrupt
    handler on proxy, which will raise on the proxied future.
    
    RB_ID=610718
  4. util-zk-test: Refine ServerCnxnFactory, add SBT project

    Chris Chen authored jenkins committed
    Problem:
    util-zk-test was missing from the SBT project definitions,
    and ServerCnxnFactory was missing a method that took a
    previously bound InetSocketAddress.
    
    Solution:
    Add project file. Refine ServerCnxnFactory.
    
    Result:
    Artifacts can be built and tests can be run using SBT.
    Less cumbersome use of ServerCnxnFactory.
    
    RB_ID=611280
  5. util-zk-test: ServerCnxnFactory for ZooKeeper 3.3 and 3.4+

    Chris Chen authored jenkins committed
    Problem:
    Integration tests construct test ZooKeeper servers using classes
    that are probably meant to be private. In ZK 3.4,
    NIOServerCnxn.Factory became NIOServerCnxnFactory which broke things.
    
    Solution:
    Provide a ServerCnxnFactory trait and a Factory that loads whichever
    class is available.
    
    Result:
    ZK integration tests that use this factory work against 3.3 and 3.4+.
    
    RB_ID=610068
  6. @luciferous

    [twitter-server] Stream admin/events page

    luciferous authored jenkins committed
    Problem
    
    The Sink exposes an iterator over Events, which we can render into an
    HTML stream (we already stream for the JSON handler), but we don't do
    this for the HTML admin page. Consequently, this requires the controller
    to fully buffer the HTML which could be potentially large.
    
    Solution
    
    Incrementally render and write HTML as we iterate over Events in the
    Sink.
    
    RB_ID=608234
  7. @luciferous

    [util-events] Java friendliness for Sink and Event

    luciferous authored jenkins committed
    Problem
    
    Using util-events from Java could be more idiomatic.
    
    Solution
    
    Create corresponding Java classes (whose names are pluralized,
    consistent with how we do this elsewhere), which proxy interactions to
    the Scala class.
    
    RB_ID=608110
Commits on Mar 16, 2015
  1. @mosesn

    util-registry: Allow /

    mosesn authored jenkins committed
    Problem
    
    There are actually a lot of strings that have / in them, like zk resolver strings, that we want to know about the /.
    
    Solution
    
    Allow /.  It should be fine with our current APIs.
    
    RB_ID=607844
  2. @mosesn

    ostrich: Provide a registry endpoint

    mosesn authored jenkins committed
    Problem
    
    We need Ostrich folks to be able to export registry data too.
    
    Solution
    
    Add a registry endpoint to the Ostrich admin server, move the code
    that can be shared between Ostrich and TwitterServer into util-registry.
    
    Also added some tests for Roster and updated the version of jackson
    while I was in here.
    
    Result
    
    Ostrich folk can export registry data.
    
    RB_ID=605258
  3. @luciferous

    [twitter-server] Add Sink admin endpoint

    luciferous authored jenkins committed
    Problem
    
    We don't have a nice interface via which we can look at events in the
    sink.
    
    Solution
    
    Create an admin page renders HTML of the sink in tabular form. Adding a
    JSON Accept header to the request returns a response in line delimited
    JSON.
    
    RB_ID=602998
  4. @mosesn

    util-core: Add `Once` for doing something exactly once

    mosesn authored jenkins committed
    Problem
    
    We often want to do something exactly once.
    
    Solution
    
    Provide a simple primitive for doing something exactly once.
    
    RB_ID=598510
Something went wrong with that request. Please try again.