Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also compare across forks.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also compare across forks.
base fork: twitter/finagle
...
head fork: twitter/finagle
Checking mergeability… Don’t worry, you can still create the pull request.
This comparison is big! We’re only showing the most recent 250 commits
Commits on Mar 03, 2015
@nshkrob nshkrob csl sbt build: Use git rev-parse instead of symbolic-ref
Problem

sbt build calls 'git symbolic-ref' which fails in a detached HEAD state. This breaks github travis builds, e.g. https://travis-ci.org/twitter/util/jobs/52793532

Solution

Use 'git rev-parse --abbrev HEAD' instead.

RB_ID=596030
1ea34e5
Commits on Mar 09, 2015
@jaked jaked finagle-core: move RequeueingFilter below BindingFactory
Problem:

In cases where Union nodes appear in a NameTree, we'd like requeues to
hit the same destination cluster rather than being distributed across
the Union. We are introducing Union nodes in place of multiple client
stacks (with external distribution across stacks), so making requeues
hit the same cluster retains the existing behavior, stats, etc.

Solution:

Move RequeueingFilter below BindingFactory. It's also necessary to move
FactoryToService (since requeues are mostly on service acquisition),
Role.prepFactory (currently used to delay closes for streaming HTTP
responses; it must go below FactoryToService), TimeoutFactory (must go
below FactoryToService to be effective), and Role.requestDraining (also
must go below FactoryToService to be effective).

We additionally add a FactoryToService and TimeoutFactory above
BindingFactory, because we want to rebind on each request (to account
for Dtab.local) and protect binding with a timeout; name binding happens
in service acquisition.

A higher-level point of view on all of this is that BindingFactory
should be at the top of the stack; the only thing outside it is tracing,
so that binding is captured in the request span.

RB_ID=589152
11f6e11
@jaked jaked finagle-core: make ClientBuilder RetryingFilter a Stack.Module and in…
…sert it down the stack

Problem:

In cases where Union nodes appear in a NameTree, we'd like
ClientBuilder-configured retries to hit the same destination cluster
rather than being distributed across the Union. We are introducing Union
nodes in place of multiple client stacks (with external distribution
across stacks), so making retries hit the same cluster retains the
existing behavior, stats, etc.

Solution:

Make the RetryingFilter in ClientBuilder a Stack.Module and insert it
just above RequeueingFilter (a separate change puts RequeueingFilter
below BindingFactory).

Along the way, add Stack.insertBefore, convert the other filters in
ClientBuilder to Stack.Modules, and add the "retries" stats scope
whether or not retries are configured (this is the only place
non-retry/requeue requests are counted).

RB_ID=589694
09b7987
@mosesn mosesn finagle-core: Add Client and Server Stack params to the registry
Problem

We want to export Stack params to the registry

Solution

Reuse StackRegistry#register to register params in the GlobalRegistry

RB_ID=593178
8055deb
@dschobel dschobel Problem
It's hard to know what's preventing the process from exiting once
you've closed all the clients/servers you thought was holding them
open.

Solution

Make the ExitGuard.guard API private since it's only used by finagle
(see Marius' comment on the linked ticket). Then add a label to the
exit guard so that we can identify which client/server is keeping
the process alive if it hasn't shut down. Users who wish to see
this can use the ExitGuard.explainGuards() method.

Result

ExitGuard.explainGuards() will give a human-readable explanation
of why the process is guarded from exiting.

RB_ID=597192
TBR=true
95dd3bd
@mosesn mosesn finagle-benchmark: {spanid,statsreceiver} caliper => jmh
Problem

We should migrate finagle's benchmarks from caliper to jmh.

Solution

Set up finagle for jmh, migrate a few simple tests

RB_ID=595572
bded787
@mariusae mariusae finagle-core, aperture balancer: allow minimum aperture sizes; defaul…
…ts by flag

Problem

1. There's no way to specify minimum aperture size (e.g., to control
load banding).

2. Parameterization of the aperture load balancer is fixed; this makes
it difficult to experiment with different parameters.

Solution

Introduce parameter minAperture so that we can have a minimum aperture
size. Add flag com.twitter.finagle.loadbalancer.apertureParam to
control defaults.

RB_ID=592374
f40aaab
@luciferous luciferous finagle-zipkin: Remove de/serialization support for BinaryAnnotation
Problem

BinaryAnnotation is a key-value pair where the type of value is `Any`.
Jackson sometimes tries to deserialize the value into a base64 encoded
byte stream. It does this even for JSON numbers.

Solution

Remove support for de/serializing BinaryAnnotation, for now.

RB_ID=597744
c9851aa
@nshkrob nshkrob finagle: Revert two RetryingFilter changes that cause retries stats t…
…o disappear

Problem

These two changes cause the retries stats to not be shown; requests are potentially not retried.

Solution

Revert until further investigation.

- Revert "finagle-core: make ClientBuilder RetryingFilter a Stack.Module and insert it down the stack"
- Revert "finagle-core: move RequeueingFilter below BindingFactory"

RB_ID=599320
TBR=true
d6a180d
@mariusae mariusae finagle-thriftmux: queue writes that happen before protocol negotiation
Problem

RB 592182 handles writes as Mux->Thrift conversions while waiting
for protocol negotiation. This introduces a race where, if we send a
control message while still undergoing negotiation, a Tdrain message
could be interpreted as a channel close event. This could make
draining less useful when rejection sessions, e.g., on startup.

Solution

Instead of converting the writes, queue them and after negotiation,
replay them through the whole pipeline, as if they were newly
originated.

RB_ID=598872
78d5588
@mariusae mariusae finagle-core: mark closed service returns as Restartable
RB_ID=599760
d9d2567
@dschobel dschobel finagle-core: stabilize ExitGuard test
Problem

An ExitGuard test is unreliable because it verifies interrupt state.

Solution

Test isAlive state instead.

RB_ID=600120
2086fca
@kevinoliver kevinoliver finagle-http(x): Fix flakey StreamingTest
Problem

Some of finagle-http(x)'s StreamingTest fail occasionally.

Solution

Fix the tests to properly sequence the client requests.

RB_ID=599686
2b54fd3
@edma2 edma2 finagle-core: Service.rescue should only catch NonFatal exceptions
Problem:

Service.rescue catches "synchronous" fatal exceptions.

Solution:

Don't catch all exceptions, just NonFatal ones.

Result:

Fatal exceptions aren't wrapped inside Futures when you call Service.rescue.

RB_ID=601006
57b68d2
@dschobel dschobel finagle-core: fix exit guard leak in integration tests
Problem

c.t.finagle.IntegrationBase creates non-daemonized clients which
are leaking exit guards because the clients aren't being closed.
This causes test flakiness in our ExitGuard tests which want to
assert things dependent on the global shared state intrinsic
to exit guards.

Solution

Daemonize the clients so we don't create the exit guards in the first
place.

Result

Less ExitGuard test flakiness.

RB_ID=601212
6bdffad
Commits on Mar 16, 2015
@jaked jaked finagle-core: require a StackTransformableClient in ClientBuilder.stack
Problem:

We would like to be able to manipulate the client stack in
ClientBuilder (in order to insert retries etc. after name binding), but
ClientBuilder can be configured with a Client which is not a StackClient
by calling stack().

Solution:

All existing Clients are already StackClients except for
ThriftMux.Client; it cannot implement StackClient in a type-safe way
because it uses different Req/Rep types externally vs. internally in its
Stack.

Instead we define a weaker StackTransformableClient which does not
expose the underlying Req/Rep types of the Stack, but permits
transformations of the stack which are indifferent to these types. This
covers the uses in ClientBuilder (and probably all the uses in Finagle).

We require a StackTransformableClient in ClientBuilder.stack(), and
implement StackTransformableClient in StackClient and ThriftMux.Client.

RB_ID=600720
4d7cf77
@jaked jaked finagle-core: move RequeueingFilter below BindingFactory
Problem:

In cases where Union nodes appear in a NameTree, we'd like requeues to
hit the same destination cluster rather than being distributed across
the Union. We are introducing Union nodes in place of multiple client
stacks (with external distribution across stacks), so making requeues
hit the same cluster retains the existing behavior, stats, etc.

Solution:

Move RequeueingFilter below BindingFactory. It's also necessary to move
FactoryToService (since requeues are mostly on service acquisition),
Role.prepFactory (currently used to delay closes for streaming HTTP
responses; it must go below FactoryToService), TimeoutFactory (must go
below FactoryToService to be effective), and Role.requestDraining (also
must go below FactoryToService to be effective).

We additionally add a FactoryToService and TimeoutFactory above
BindingFactory, because we want to rebind on each request (to account
for Dtab.local) and protect binding with a timeout; name binding happens
in service acquisition.

A higher-level point of view on all of this is that BindingFactory
should be at the top of the stack; the only thing outside it is tracing,
so that binding is captured in the request span.

RB_ID=600920
0bc4217
@jaked jaked finagle-core: make ClientBuilder RetryingFilter a Stack.Module and in…
…sert it down the stack

Problem:

In cases where Union nodes appear in a NameTree, we'd like
ClientBuilder-configured retries to hit the same destination cluster
rather than being distributed across the Union. We are introducing Union
nodes in place of multiple client stacks (with external distribution
across stacks), so making retries hit the same cluster retains the
existing behavior, stats, etc.

Solution:

Make the RetryingFilter in ClientBuilder a Stack.Module and insert it
just above RequeueingFilter (a separate change puts RequeueingFilter
below BindingFactory).

Along the way, add Stack.insertBefore, convert the other filters in
ClientBuilder to Stack.Modules, and add the "retries" stats scope
whether or not retries are configured (this is the only place
non-retry/requeue requests are counted).

RB_ID=600760
cefdf89
@mosesn mosesn finagle-mux: Treat closed connections as successful closes
Problem

There's a race where a connection gets closed, but a server has
already selected it for server dispatching.  Normally this would
be OK, but Future.join relies upon every Future being successful
to not be satisfied, since a failure short-circuits it, and we're
relying on the joined futures being alive to keep TwitterServer
alive.  After the future has been satisfied, the program exits,
shutting down all of the sockets without waiting for the graceful
shutdown to finish, even if there's time remaining.

Solution

Turn that frown upside down!  ChannelClosedExceptions are now
successful close results.

Result

TwitterServer users of graceful shutdowns can handle racy client
hangups.

RB_ID=601282
9ff8e54
@takei-shg takei-shg add SmartNews to the Adopters list
Signed-off-by: Travis Brown <tbrown@twitter.com>

RB_ID=603332
13154f5
@kevinoliver kevinoliver finagle-stats: Do not read format flag eagerly
Problem

The global flag, `-com.twitter.finagle.stats.format` was
being read before flag parsing had finished leading to
the wrong formatter used at runtime.

Also, 0.95 percentile was not being formatted as p95.

Solution

Read this flag as needed and render p95 properly.

RB_ID=601486
0b960fd
@jaked jaked finagle-core: fix typing of StackClient.configured
Problem:

StackClient.configured is a StackTransformableClient, not a StackClient.

Solution:

Override the configured methods to the more specific types. Along the
way, rename StackTransformableClient to StackBasedClient, and clean up
the typing: separate the transform method into a Stack.Transformable
trait, and where we need more-specific types for Parameterized and
Transformable methods, mix in more-specific instantiations of the
traits.

RB_ID=603550
fe95157
@kevinoliver kevinoliver finagle-stats: Fix latched counter deltas
Problem

Latched counters were wildly wrong depending on what time
their values were queried.

Solution

Be more Ostrich like and calculate the deltas for the each
time period at update time. The current absolute value is
also kept in order to compute the delta for the next update.

Result

Ostrich compatibility mode is more "Ostrichy".

RB_ID=600710
de932e4
@edwelker edwelker add NCBI to adopters list
Signed-off-by: Travis Brown <tbrown@twitter.com>

RB_ID=603898
f65655c
@mosesn mosesn util-core: Add `Once` for doing something exactly once
Problem

We often want to do something exactly once.

Solution

Provide a simple primitive for doing something exactly once.

RB_ID=598510
478e81e
@luciferous luciferous [twitter-server] Add Sink admin endpoint
Problem

We don't have a nice interface via which we can look at events in the
sink.

Solution

Create an admin page renders HTML of the sink in tabular form. Adding a
JSON Accept header to the request returns a response in line delimited
JSON.

RB_ID=602998
a4befe1
Glen D Sanford [finagle-thriftmux] fix dtab propagation for TTwitter clients of Thr…
…iftMux servers

 Problem

 Local Dtabs are not propagated properly from TTwitter thrift clients to ThriftMux servers.

 Solution

 In TTwitterToMux, unpack delegations from the header and stuff them into a dtab.

 (note: this isn't a problem for the framed codec, which has no facility to pass delegations at all)

RB_ID=601656
1b5573f
@jaked jaked finagle-core: fix typing of StackClient.configured, again
Problem:

StackClient.configured is a StackBasedClient, not a StackClient, from
Java.

Mixing in Stack.Parameterized[StackClient] is not enough from Java. You
get

  public interface StackClient<Req, Rep>
    extends StackBasedClient<Req, Rep>, Stack$Parameterized<StackClient<Req, Rep>>

where both parent interfaces define configured() but at different
types. It seems that whichever interface comes first wins, which seems
like a scalac bug, but I'm unable to reproduce this so far in a small
example---scalac rightfully complains of illegal inheritance.

Solution:

Override the configured methods to the more specific types, add a Java
compilation test.

RB_ID=605182
b5de66e
@kevinoliver kevinoliver finagle-{stats,ostrich4}: Use httpx instead of http
Problem

We are migrating to finagle-httpx from finagle-http and
as part of that we need to fix our internal dependencies
to avoid using finagle-http.

Solution

Switch to using finagle-httpx and have twitter-server serve
the metrics endpoints properly as well.

twitter-server now depends on both finagle-http and -httpx
as a bridge until it is migrated to just using httpx.

RB_ID=604718
32577dc
@kevinoliver kevinoliver finagle-thriftmux: Improve Java compatibility for ThriftMux.server
Problem

Java cannot see `ThriftMux.server` as having the proper types
it needs to be used in `ServerBuilder.stack`.

Solution

Introduce `StackBasedServer` that mirrors `StackBasedClient`
and is also usable from Java.

RB_ID=603476
f607f17
@atollena atollena finagle-core: Expose latency compensation as a Stack.Param
Problem

LatencyCompensation operates by adding a compensation to the timeout
parameters for Transporter and TimeoutFilter. This prevents other
modules from applying compensation.

Solution

Expose the compensation as a Param, and read it in Transporter and
TimeoutFilter.

Result

New modules can apply latency compensation if needed.

RB_ID=603102
d001401
@vkostyukov vkostyukov finagle-core: Fix early channel closure with TLS enabled
RB_ID=604786
6113a89
@csaltos csaltos Adding Talenteca to the adopters page
Signed-off-by: Travis Brown <tbrown@twitter.com>

RB_ID=607414
9223c18
@mosesn mosesn finagle-core: Register LoadService entries
Problem

It should be easy to see which classes are being service-loaded,
like metrics or ostrich stats.

Solution

Register LoadService entries in the R* Registry.

RB_ID=607690
4f514f6
@mosesn mosesn util-registry: Allow /
Problem

There are actually a lot of strings that have / in them, like zk resolver strings, that we want to know about the /.

Solution

Allow /.  It should be fine with our current APIs.

RB_ID=607844
7961eef
Glen D Sanford [finagle-thrift[mux]] add RichRequestHeader
 Problem

 There's duplicate code between c.t.f.thrift and c.t.f.thriftmux for
 pulling things out of thrift RequestHeaders. It would be nice to
 centralize it in case we find bugs or need to add things later.

 Solution

 * Add RichRequestHeader value class, with methods for clientId, dest,
   dtab, and traceId.
 * Change c.t.f.thrift.TTwitterServerFilter and c.t.f.thriftmux.Netty3
   to use the new class.
 * Fix a Buf deprecation warning while I'm in here
 * Fix a missing BUILD dependency

 Result

 Less duplicate code

RB_ID=607838
ecfa3d1
@dschobel dschobel finagle-thrift: fix load-balancer assumption in test
Problem

"finagle client vs. synchronous thrift server should talk to multiple servers" asserts
that a cluster of N hosts will each receive a single request when N requests are dispatched.
This assertion does not hold for any load balancer other than the HeapBalancer.

Solution

Test balancer flag and ignore non-heap configurations.

RB_ID=602678
b53494e
Commits on Mar 23, 2015
@dschobel dschobel finagle-kestrelx: cherry pick MultiReader patch from finagle-kestrel
cherry pick https://reviewboard.twitter.biz/r/461785/ onto kestrelx

RB_ID=609460
981fb61
@stevegury stevegury finagle-mux: GCA use com.twitter.hotspot.System.minorGc instead of Sy…
…stem.minorGc

Problem

The next release of the JVM will remove the API System.{minor,full}Gc in
favor of com.twitter.hotspot.System.minorGc

Solution
Update the usage in GC-Avoidance (GCA).

RB_ID=556948
96f039f
@mosesn mosesn finagle-httpx-compat: Rename artifact
RB_ID=609848
4503b9c
@mosesn mosesn finagle-core: Handle broken pipes correctly
Problem

We don't handle sigpipe nicely.

Solution

Turn java.io.IOException("Broken pipe") into a ChannelClosedException.

RB_ID=610632
b8adec4
@penland365 penland365 Problem
finagle-redis's tests use the specs framework, which is not available for Scala
2.11. Their structure is also not ideal.

Solution

Migrate all tests from specs to ScalaTest, restructuring as appropriate based on
conversations with Moses Nakamura and others, and adding FLUSHALL support to
facilitate integration testing. Parts of the test migration were completed by
Travis Brown.

Result

Finagle no longer has any test dependencies on specs.

RB_ID=607602
377e7ad
Chris Chen Update ZooKeeper integration tests to use util-zk-test
Problem:
ZooKeeper integration tests use NIOServerCnxn.Factory to construct
in-process servers, but in 3.4 that changed to NIOServerCnxnFactory,
which breaks tests.

Solution:
To make it easier to upgrade ZK in science and birdcage, the
problematic class has been placed behind ServerCnxnFactory in
util-zk-test. This change updates the following test suites to use
that factory:

birdcage/finagle/finagle-memcached
birdcage/finagle/finagle-memcachedx
birdcage/finagle/finagle-serversets
birdcage/kestrel/server
birdcage/socialgraph/repair
birdcage/twitter-server-internal
birdcage/twitter-zookeeper/client-serverset2
birdcage/twitter-zookeeper/tests/util

Result:
ZK can be upgraded in birdcage from 3.3 to 3.4 without breaking tests.

RB_ID=610862
b141248
@travisbrown travisbrown Remove unused finagle-protobuf project
Problem

finagle-protobuf is not being compiled, tested, or used at Twitter and is badly
out of date.

Solution

I've created a new finagle-protobuf project in the Finagle organization on
GitHub from the subproject here (after rebasing #91). This commit completes the
move by deleting the unused subproject and removing the commented-out build
config.

Result

finagle-protobuf lives in the Finagle organization.

RB_ID=611556
6a73425
@luciferous luciferous [util] Use proxyTo instead of become
Problem

In the case where it is passed a promise, Promise.become merges the
listeners of two promises, an operation which requires modifying the
internal state of both Promises. The complication arises when the
promises each have an interrupt handler: one gets overwritten.

Consider the following, which demonstrates the behavior of `become` on
promises with interrupt handlers.

    val a, b = new Promise[Unit]
    a.setInterruptHandler { case _ => println("A") }
    b.become(a)
    b.setInterruptHandler { case _ => println("B") }
    a.raise(new Exception)

This prints "B", the action in the interrupt handler for `b`, which is
unexpected because we raised on `a`.

Solution

A cleaner solution is to use proxyTo, which does not merge listeners,
but relies only on the standard Future.respond. We can set an interrupt
handler on proxy, which will raise on the proxied future.

RB_ID=610718
819e034
Joy Su remove elin from OWNERS files
add aradeski to tweet scheduling, remove tweetypie engineers

RB_ID=613682
83f5b18
@blackicewei blackicewei finagle-memcached: stack based pipelining finagle-memcache client
Problem

`KetamaClientBuilder` does not use pipelining, and is based on ClientBuilder.

Solution
add a Stack based Ketama client builder which uses pipelining.

RB_ID=571806
TBR=true
4692ebc
@stevegury stevegury finagle-mux: FailureDetector closes connections on long inactivity
Problem

When a network partition occurs, the connection used by Mux may never be closed.
This is usually not a problem because Timeout/FailFast/FailureAccrual and others
will mark the status of the ServiceFactory as Busy. But for clients configured
with only one server, this never closes the connection.

Solution

Modify the FailureDetector, to close the underlying connection after a
(configurable) period of inactivity (no response to ping).
This period is configured as a multiple of the moving average of the ping latency.
By default, this is disabled but I recommend using a pretty large value (in the
range of 1000).

Modifications

I also introduced a new stats "failuredetector/ping_latency_us" which expose an
histogram of the ping latency in microseconds.

RB_ID=613666
e724f19
@mosesn mosesn macaw-framework: Use raw finagle types
Problem

Macaw servers don't support streaming.  The netty / finagle converter
kills streaming by accident.

Solution

Use ServerBuilder until macaw is on httpx, where the Http.serve
will use the finagle types natively.  Also, change RequestProxy to
proxy reader and writer.

RB_ID=613120
9bcde40
Commits on Mar 30, 2015
@dschobel dschobel finagle-core: make FailedFastException a WriteException
Problem

FailedFastExceptions are guaranteed to not be written to the network
since they only arise in response to connection failures. This makes
them WriteExceptions and safe to re-enqueue.

Solution

Categorize FailFastException as WriteExceptions.

Result

New behavior:

  In the case where all hosts are down*: zero requeues, failed request.

  In the case of a race where one host is failed-fast after
    load balancer selection: potentially saved request via requeueing.

Old behavior:

  Failed request in both cases.

*or any other component of the stack beneath the requeue filter has
  status < Open

RB_ID=569636
dec7600
@spockz spockz Made RetryingFilter extendable.
Problem

The main logic of the RetryingFilter is good enough for me, however it does not
pass the request to the policy. In my case I always want to retry GET requests.
Therefore, I want to extend the RetryingFilter such that the new apply() method
can use the request to create a new policy. In the current situation that is not
possible because the dispatch method is private. Consequently, the
RetryingFilter is not extendable.

Solution

Changed the access modifier from private to protected.

Result

Classes that extend the RetryingFilter can now call dispatch.

Signed-off-by: Steve Gury <stevegury@twitter.com>

RB_ID=615154
24d6e3e
@luciferous luciferous Upgrade to netty-3.10.1.Final
RB_ID=615714
TBR=true
e24642d
@mariusae mariusae finagle-core: eliminate some allocations in context processing
Problem

The deadline branch introduced a standard context that is now shipped
in basically all clients and servers. This resulted in a relatively steep
allocation regression in the mux smoke tests (~20%). With this change,
we recoup all of it.

Solution

Reduce allocation during key processing and comparison:

1. Make sure context keys are ByteBuffers so that we can;
2. compare them directly without copying them.

On master, we get this in the Mux allocation test

	before 8552 bytes/iter
	after 18486 bytes/iter

The effect is (much) more pronounced in the deadlines branch.

RB_ID=606326
TBR=true
4caf457
@roanta roanta finagle-benchmark: Add JMH benchmarks for Balancers
RB_ID=617996
0f4af68
@roanta roanta finagle-benchmark: Convert Pool Benchmark to JMH
RB_ID=616930
3cdd008
Chris Chen 3rdparty: Update zookeeper to 3.4.6, zookeeper-testing to match.
Problem:
Birdcage has been stuck at ZooKeeper 3.3.4 for years. In order to
finish the upgrade, zookeeper-testing from twitter.common needs to
be upgraded in lockstep.

Solution:
Upgrade zookeeper-testing and ZooKeeper 3rdparty deps at the same
time.

Result:
Science and birdcage both use ZooKeeper 3.4.6 as the base 3rdparty
ZooKeeper.

RB_ID=617360
0a18b24
@atollena atollena [finagle-http] trivial typo fix in finagle-httpx publishing
RB_ID=620392
4e0ce43
@roanta roanta finagle-benchmark: more jmh less caliper
RB_ID=619028
6f07f60
Commits on Apr 06, 2015
@kevinoliver kevinoliver finagle-stats: Alternative histogram implementation
Problem

Given the minimal space constraints the Commons Metrics histograms
do not have a high enough guarantee on their error. The epsilon is
currently 0.021 which means that the 99.99% can only be guaranteed to
be a 97.89%. While in practice it is typically much tighter than this,
we prefer tighter guarantees.

Solution

Use a bucketing algorithm that with configurable error bounds, E,
between 0 and 1, such that that for any given percentile the value
will be at most (1 + E) percent off from the actual percentile.

Result

An implementation heavily based on Ostrich's `Histogram` while
fairly tight error bounds and a memory footprint of ~29kb per `Stat`.

RB_ID=616918
8a95512
@dschobel dschobel finagle-core: HeapBalancer's status calculation should discard dummy …
…node

Problem

HeapBalancer calculates status inclusive of its dummy head node, a
node which has perpetually Open status. This results in the
HeapBalancer's status being perpetually Open.

Solution

Discard the head node when calculating status.

Result

Higher fidelity cluster status for HeapBalancer and by extension
fewer requeues when they have a poor chance of success. We also
gain consistency in status semantics between heap, choice and
aperture balancers for empty and entirely unavailable clusters.

RB_ID=620638
9a4c2f5
@kevinoliver kevinoliver Problem
Scala creates Functions instead of a PartialFunction when used
anonymously as in:

  Future.onFailure { case NonFatal(e) => ... }

This meant that the special handling introduced in RB_ID=588992 failed
to actually help. The unit test "lied" because it passed explicit
PartialFunctions instead of the more idiomatic anonymous ones.

Solution

Revert the changes introduced in RB_ID=588992 and fix the call sites
that were using it incorrectly.

RB_ID=620938
5304064
@mariusae mariusae finagle-core: introduce a deadline context
Problem

We'd like to communicate deadlines across process boundaries so that
we can use this information to implement better queueing policies.
Furthermore, even if there isn't an available deadline, we'd like to
know the walltime on the sending host so that we can understand IPC
queueing times, and implement load conditioning when hosts are
overloaded.

Solution

Introduce a deadline context, com.twitter.finagle.Deadline. This will
propagate through all prototols that support contexts.

RB_ID=601720
02704ce
@mariusae mariusae finagle-mux: rewrite server dispatcher
Problem

My previous change to add metrics made it very obvious that
the Mux server dispatcher was becoming increasingly difficult
to reason about, especially in terms of session state.

Solution

Rewrite the server dispatcher in a more sustainable way. Indeed
this process found more races. Among the changes are:

 *	Separate out request processing from session management
 *	Remove unnecessary configuration (canDispatch, ping), and
	move their functionality into a filter chain.
 *	Introduce a separate transaction tracker that manages draining.
 *	Introduce a fine-grained session state so that we can tell the
	difference between draining, client, and server hangups.

This also introduces 4 new metrics to track draining stats:

**draining**
  the number of times the server has initiated session draining

**drained**
  the number of times the server has succesfully completed the draining protocol within its allotted time

**clienthangup**
  the number of times sessions have been abruptly terminated by the client

**serverhangup**
  the number of times sessions have been abruptly terminated by the server

Result

Much easier to reason about the server dispatcher code and session states.

Smokestack

old:
	mux	1000000	16555 bytes/iter
	muxsess	1000000	30348 bytes/iter

new:
	mux	1000000	16867 bytes/iter
	muxsess	1000000	31555 bytes/iter

RB_ID=580754
dc6bd22
@luciferous luciferous [finagle-core] Fail readq before close
Problem

Failures sent upstream from netty (via handleUpstream), don't fail the
readq quickly enough. The fail calls close(), which hands execution back
to netty (via Channels.close).

We observe in one case that after sending a failure, netty is still able
to send a subsequent message, which ends up on the readq, before we fail
it.

The related issue is that Netty sends upstream a partially deserialized
and corrupt HttpResponse after sending failure.  The ChannelTransport
should, upon receiving this failure, set the readq to Failing, which
causes rejection of subsequent offers. But because of the order in which
we call readq and close, the corrupt HttpResponse makes it into the
readq before it is set to Failing.

Solution

We fail the readq before close. This causes readq to reject any offers
after the receipt of the first failure.

Now, when Netty sends a failure and then the corrupted Response, the
client dispatcher fails instead of passing the bad Response back to the
user.

RB_ID=624140
817c258
@roanta roanta finagle-benchmark: convert thrift benchmarks to jmh / smokestack
RB_ID=623478
812613a
@dschobel dschobel birdcage/*: migrate to finagle-kestrelx
Migrate birdcage from finagle-kestrel to finagle-kestrelx as part
of Twitter's netty4 migration.

RB_ID=618432
c6bbbfe
@stuhood stuhood birdcage: Fix finagle-kestrelx provides clause
- Fix finagle-kestrelx provides clause

http://static-ci.twitter.biz/publish-birdcage-snapshot/2015-04-03_13-00-00/6/console.html

RB_ID=625956
TBR=true
f11e534
Glen D Sanford remove glen from various OWNERS files
RB_ID=625782
TBR=true
59c7f40
@travisbrown travisbrown Update documentation about contributing
Problem

The contributing documentation currently recommends submitting pull requests
against master.

Solution

Update the contributing documentation to reflect the current workflow, and add
a note about branches to the main README.

Signed-off-by: Travis Brown <tbrown@twitter.com>

RB_ID=623652
4cbd42d
@kevinoliver kevinoliver util-stats: Fix deadlock in CumulativeGauge
Problem

Lock acquisition order was not always consistent between
`CumulativeGauge` and `StatsReceiverWithCumulativeGauges` which can
lead to deadlocks.

Solution

Remove synchronization from `StatsReceiverWithCumulativeGauges`.

RB_ID=624714
7431f03
@travisbrown travisbrown Fix problems with SBT and Travis CI builds, update Scala and SBT vers…
…ions

Problem

SBT builds for Finagle and associated projects were failing for a number of
reasons, including third-party dependency versions that had been updated for
Pants but not SBT, new tests that are run differently by the SBT test runner,
and changes to JSON serialization that don't work the same on Scala 2.10 and
2.11. Our SBT and Scala versions were also out of date.

Also Travis CI's container-based infrastructure currently has a bug that causes
the build to crash when getting the local hostname with IPv4 on OpenJDK 7
(reported in #3120 on GitHub). Moving back to the standard Travis infrastructure
is a trivial change, so I've included it here.

Solution

I've updated the SBT and Scala versions throughout, and have updated third-party
dependency versions to match the configuration in 3rdparty (note that I've
followed the last update to scrooge/project/build.properties in not updating the
custom SBT script for Scrooge). TwitterServer now uses scala.xml, which is a
separate module in Scala 2.11, so I've added a conditional dependency in the SBT
build.

The new ExitGuard tests were failing on SBT because clients were being left open
in other test suites. I've daemonized all of these clients so that they don't
create exit guards.

In JsonSinkTest in TwitterServer I've desugared an implicit conversion
provided by a view bound because Scala 2.11 prohibits Object as the target of an
implicit conversion.

Because Jackson doesn't correctly serialize Scala case classes that are defined
locally in vals on Scala 2.11, I've moved three case classes out of vals and
into the enclosing objects, changing names where necessary to avoid collisions,
and making each case class private to the object.

I've also switched all projects with a custom SBT script back to the standard
Travis CI infrastructure.

Result

We're on the most recent versions of Scala and SBT, the SBT builds actually
work, and Travis CI won't fail spuriously on OpenJDK 7.

RB_ID=619988
af37ec9
@travisbrown travisbrown Use sbt-unidoc plugin and remove out-of-date copy-pasted version
Problem

A couple of months ago I set up our SBT builds to include API doc links in the
POMs, which makes it possible for e.g. Finagle's API docs to link to Util's.
This didn't actually work because we were using an old copy-pasted version of
Unidoc, and unfortunately the same bug was present in the sbt-unidoc plugin.

Solution

My fix for the sbt-unidoc bug is now available in the 0.3.2 release, so we can
switch to using it. I've also added an API doc link to TwitterServer's SBT
config, even though historically we haven't published TwitterServer's API docs
(I'd like to start).

Result

We can build Finagle's API docs with Unidoc and have references to classes from
Util and the Scala standard library automatically linked to their definitions in
those projects' API docs.

RB_ID=627082
feb3c5e
Commits on Apr 13, 2015
@mariusae mariusae finagle-core: maintain stats for transit times and deadline budgets
This will provide some helpful production numbers to tune a
threshold-based admissions controller.

RB_ID=627580
d541441
@kachayev kachayev Updated list of adopters with Attendify
* keeping alphabetical order

* link to the main page (there is no appropriate blog post just now)

Signed-off-by: Travis Brown <tbrown@twitter.com>

RB_ID=628550
46320e1
Charith Ellawala Add support for Redis HyperLogLog commands
Problem

Finagle-redis currently lacks support for HyperLogLog commands
(http://redis.io/commands#hyperloglog).

Solution

Add code to enable calling PFADD, PFCOUNT and PFMERGE from RedisClient

Result

RedisClient API will have three new methods:
- pfAdd
- pfCount
- pfMerge

Signed-off-by: Travis Brown <tbrown@twitter.com>

RB_ID=627512
036c693
@keeth keeth Add Spright to Finagle adopters list
Signed-off-by: Travis Brown <tbrown@twitter.com>

RB_ID=629782
96450e4
@dschobel dschobel finagle-memcached[|x]: fix race in c.t.f.memcached.integration.Memcac…
…hedTest

Problem

Test "re-hash when a bad host is ejected" is non-deterministically
failing in asserting the eviction of a failed client.

Solution

Lower the failure threshold and block on cache #set in test.

RB_ID=628512
f6ce8cd
@dschobel dschobel finagle-core: fix flakyiness in c.t.finagle.builder.EndToEndTest
Problem

Two c.t.finagle.builder.EndToEndTest tests are sporadically failing
with timeout exceptions.

Solution

Remove timeouts.

RB_ID=631366
d4595fe
@kevinoliver kevinoliver util/finagle: Small changes to avoid allocations
Problem

In various places we use methods that rely on Scala's implicits to
enrich them. This typically at the cost of an unnecessary object
allocation. There are some other simple changes to be made that
avoid allocations as well.

Solution

Use methods that do not require allocations. Some examples:

* String.nonEmpty => !String.isEmpty
* String.size => String.length
* StringBuilder.size => StringBuilder.length
* Array.size => Array.length
* Seq.empty => Nil

RB_ID=630964
09d490f
@mosesn mosesn finagle-core: Doesn't create negative stats
Problem

Ostrich logs whenever someone sends it a negative value, and because
of clock skew, we often have negative values when comparing timestamps
between machines.

Solution

Replace negative values with 0.

RB_ID=631574
c953a94
@luciferous luciferous [trivial] Add withStreaming to Httpx protocol object
Problem

The other params (e.g. MaxRequestSize) have with-helpers (e.g.
`withMaxRequestSize()`) to help configure the parameter. The
Streaming param is missing a with-helper.

Solution

Add `withStreaming()`.

Result

Users can write `Httpx.server.withStreaming(true)` instead of
`Httpx.server.configured(Httpx.param.Streaming(true))`. Also
the Streaming param follows the same convention as the other
Httpx params.

RB_ID=631164
3b7e62c
Commits on Apr 20, 2015
@mosesn mosesn finagle-core: Removes old registry entries when they're no longer ref…
…erenced

Problem

Although creating many finagle clients is typically an antipattern,
it's occasionally necessary, for example for web crawlers or link
shorteners.

Solution

We remove old registry entries when they're no longer referenced.
This requires adding a \`remove\` method to Registry, and hooking
it up to closing the ServiceFactory you're handed.

Result

As long as users close their resources, registry "just works".

RB_ID=625826
1339f82
@vkostyukov vkostyukov finagle-mux: Decrease severity of a noisy logger in mux client
RB_ID=633826
f5724e9
@dschobel dschobel finagle-memcached[|x]: flaky MigrationClient and ReplicationClient in…
…tegration tests

Tag flaky Migration and Replication client tests as flaky and mark
for low-priority follow up.

RB_ID=634214
93dcce1
@vkostyukov vkostyukov finagle-http(x): Improve HttpMuxer routing
RB_ID=634438
40993a3
Gelin Zhou finagle-core: adding Filter.choose to dynamically choose a Filter bas…
…ed on incoming requests

Problem

It is common that we only want to apply a Filter to certain requests based on some criterion.
This RB adds a new function Filter.choose() that avoids mixing filtering logic and predicate logic.

Solution

Adding Filter.choose [Req, Rep]  (pf: PartialFunction[Req, Filter[Req, Rep, Req, Rep]]) to separate out filtering logic from the predicate logic. The partial function chooses a Filter to apply based on incoming requests, or does not apply any Filter if the PartialFunction pf is not defined at the request.

RB_ID=630022
3c1025e
@travisbrown travisbrown Update SBT's ScalaCheck and ScalaTest versions to match 3rdparty
RB_ID=627742
3e75251
@nshkrob nshkrob scrooge: Reduce allocations in generated code
Problem

Scrooge-generated code allocates reusable growable byte arrays for the response bytes up to a fixed size. If the size exceedes the max size, the array is discarded. For services with large response sizes, this leads to a large allocation for every response.

Solution

Make the max buffer size tunable.

- Add a maxThriftBufferSize parameter to the server constructor and a Stack param for it.
- Add a srv/thrift/buffer/resetCount stat.
- Increase the default maxCapacity of the reusable buffer in $FinagleService from 4K to 16K.
- Replace orElse with pattern matching to avoid allocating a Function1.

Result

133124 bytes/iter -\> 102131 bytes/iter in allocation testing.

RB_ID=628710
TBR=true
effd91e
@atollena atollena [finagle-core] Fix Path and NameTree grammar descrition
Problem:

The grammars described in NameTree and Path do not match the parser.

- Path rejects '/' but it's explicitly listed as a valid path

- NameTree weights do not support full c-style float syntax.

Also the parser blows up with weights composed of a single '.'.

Solution:

Make the grammar description match the parser, fix the parser on
. weights.

RB_ID=637410
37e4e9c
@kevinoliver kevinoliver finagle-memcache(x): Mark more integration tests as flaky
Problem

The flaky tests continue to be found.

Solution

Mark them as flaky until they can be addressed holistically.

RB_ID=637974
b4c918c
@edouardswiac edouardswiac [classpath isolation] merge dependencies that share the same source i…
…n finagle-mysql tests

RB_ID=638728
3de3491
@vkostyukov vkostyukov finagle-zipkin: Add name to the RawZipkinTracer shutdown hook thread
RB_ID=639186
45dea7d
@luciferous luciferous [trivial] Display Trace and Span IDs in hexadecimal
Problem

TraceId and SpanId values are displayed as Longs. They are more readable
as hexadecimal, which is the conventional presentation.

Solution

Display TraceId and SpanId values as hexadecimal.

RB_ID=639192
bf87f05
Commits on Apr 22, 2015
@mosesn mosesn finagle-exp: Make BackupRequestFilter allocate less
Problem

BackupRequestFilter was allocating right, left, and center.

Solution

Remove as many allocations as possible while maintaining correctness
and clarity.

RB_ID=634734
e12c940
@nepthar nepthar finagle-core: Surface TLS peer cert info per request via Transport
Problem

While it's possible to enable 2-way TLS, there's no way of accessing any details of the SSL session on a per-request basis, which is useful client/server identification and auth.

Solution

SSL Session information is accessible via netty's SSLHandler, which is added to the ChannelPipeline for secure connections. This adds an optional SSLSession data member to the Transport trait. If present, this is propagated to each call to the service through Contexts.local via a dispatcher.

Result

SSL session info (if present) can be obtained by calling Transport.sslSession().

RB_ID=631870
3d6f236
@roanta roanta finagle-core: Move finagle-mux requeues into the default stack
Problem

The requeue module in finagle-core lacked support for service
acquisition failures under certain conditions. The requeue
module in mux supports this plus requeues are credited at a ratio
of requests.

Solution

Move the mux requeue module into the default client stack.

RB_ID=627412
38dff0f
@kevinoliver kevinoliver finagle-core: Deal with flakey tests
Problem

A few tests are being noticed as flakey in CI.

Solution

Mark some of the tests as flakey given the current situation.  Mixin
IntegrationPatience as needed as well.

RB_ID=640212
4193264
@mosesn mosesn csl: Publish externally with pants, sbt, and friends
Problem

It's difficult to publish util, finagle, etc. because we haven't
updated them for the new maven-less world.

Solution

Scripts that make it easy to publish with sbt and pants.  Bumped
the server-set version because we stopped publishing thrift
externally in ci.  Changed an ExitGuard test which broke under
weird sbt conditions.

Result

We're in a position where we can start publishing finagle, etc
again!

RB_ID=633570
7890917
@roanta roanta finagle: add roanta as a top-level owner
RB_ID=641192
a2cf274
@tianxiao-ma tianxiao-ma finagle-core: Use UTF-8 charset in LoadService
Problem

LoadService uses the platform's default charset
when reading files which can be incorrect.

Solution

Always use UTF-8.

RB_ID=641262
aad7344
@luciferous luciferous [finagle] Cleanup compile warnings
Problem

There are lots of compile warnings in finagle.

Solution

Fix some of them.

RB_ID=640914
ac8b6eb
@kevinoliver kevinoliver finagle-core: Handle when DefaultTracer.self is null
Problem

In a situation where a `Logger` depends on Tracing but the Tracer has
not yet been initialized, `DefaultTracer.self` can be null.

Solution

Add defensive null checks in `DefaultTracer`.

RB_ID=639874
TBR=true
f308d9f
@atollena atollena [finagle-core] Namer abstract class for java compatibility
Problem

It is difficult to specialize Namer from Java because traits are
converted to interfaces, and we lose the implementation of bind and
bindAndEval.

Solution

Provide an abstract class AbstractNamer that can be inherited from
Java.

RB_ID=641952
625793a
Commits on Apr 27, 2015
@atollena atollena [finagle-http] Add an assertion for cookie removal
Problem

We don't test for header removal when a cookie is removed.

Solution

Add an assertion.

RB_ID=644112
ccb74ae
@missingfaktor missingfaktor finagle-mysql: Make PreparedStatment parameters type safe.
Problem

In the current Finagle MySQL, PreparedStatement#apply has a type
Seq[Any] => Future[Result]. This means that one might accidentally
end up passing things that cannot be written, and be in for surprises.
(Happened to us quite a few times now.) This might be revealed in
the warnings and the tests, but we can do much better: We can
eliminate such errors at compile time itself.

Solution

Change PreparedToStatement#apply to have type Seq[Parameter] =>
Future[Result]. Parameter is a class that makes use of existential
types and packs a value A and an implicit instance CanBeParameter[A]
together. CanBeParameter is a type-class. If you have an instance
of CanBeParameter[A] available, that serves as an evidence that,
well, a value of A can be a used as a parameter.

This PR ensures that all the types that were supported before are
still supported the same way.

null required special handling in a couple of places:

Parameter.wrap doesn't kick in for null because null is already a
member of Parameter type. So we need to sanitize the Seq[Parameter]
in ExecuteRequest#apply, and replace all nulls with NullParameter.
If there is a null with a static type A for which there is a
CanBeParameter[A] evidence available, then we might end up with,
for example, a stringCanBeParameter instance (if A is String) and
typeCode etc will yield wrong values. To avoid this, Parameter.wrap
has to guard for null and return a NullParameter in that case.  This
PR retains the support for null as is, for backwards compatibility.

Going forward, we could stop supporting null altogether and avoid
the complications I talked about before. We could instead support
Option which is a more principled way to deal with these sort of
things.

RB_ID=636592
TBR=true
536a630
@dschobel dschobel cuckoo/*: port to memcachedx and httpx
Port cuckoo/* to finagle-memcachedx and finagle-httpx.

RB_ID=639516
4f9233b
@mosesn mosesn finagle: Report protocol to /admin/registry.json
Problem

Protocols aren't reported to /admin/registry.json, despite them
being an interesting piece of runtime configuration data.

Solution

Report protocols as part of the ClientRegistry export, in the
hierarchical name.

RB_ID=640896
c5382b3
@mosesn mosesn myriad cleanup
Cleaned up a bunch of build warnings, added Try.getOrElse

RB_ID=643354
c8d4e06
@n8han n8han Adding Hopper to Finagle adopters
Hopper uses Finagle for almost all services and service clients, including its
mobile app API.

Signed-off-by: Travis Brown <tbrown@twitter.com>

RB_ID=644458
677eb44
@mosesn mosesn finagle-serversets: Exclude slf4j-log4j12 explicitly
Problem

Zookeeper brings in slf4j-log4j12, which causes your program to
break if you also depend on log4j-over-slf4j.

Solution

Remove it.  If you depend on finagle-serversets, you can choose for
yourself.

RB_ID=641898
99d80b1
Chris Chen finagle-serversets: Memoize resolutions; Stop state leakage during no…
…de fetch.

Problem:
When using serverset2 to resolve multiple endpoints from a single serverset,
duplicate sets of endpoints are maintained. In addition, Activity.collect
leaks intermediate state so downstream computations are triggered before the
entire collection is complete.

Solution:
Memoize on (discoverer, path) so all endpoint resolutions share the same
underlying data, reducing size usage. Use Future.collect on immutable node
data so downstream computations are triggered a constant number of times
rather than linearly to the size of the underlying serverset.

Result:
Resolution is far faster.

RB_ID=644726
f9da5d9
@mosesn mosesn Revert "finagle-exp: Make BackupRequestFilter allocate less"
RB_ID=645748
1b9fe8d
@nshkrob nshkrob finagle: Reduce allocations
Problem

Unnecessary allocations in finagle-serversets and ConstFuture.

Solution

- Make counters persistent in finagle-serverset2.
- Make ZkConversion-related types value classes.
- Override ConstFuture.isDone.

RB_ID=643270
ab224ee
Chris Chen finagle-serversets: Reduce allocations during serverset changes
Problem:
When a serverset changes, we attempt to do the minimum amount of work
possible, but we create strings and use sets where unneeded.

Solution:
Check if an endpoint's data is cached before calling Buf.Utf8's
deconstructor. Preserve endpoints (path, data) as a sequence; The
AsyncInetResolver will collapse it into a Set.

Result:
Allocations during serverset changes are greatly reduced, reducing
memory pressure.

RB_ID=645180
45da771
@zdavep zdavep Add SoFi to Adopters list
Signed-off-by: Travis Brown <tbrown@twitter.com>

RB_ID=645616
3fc7971
Commits on Apr 30, 2015
@nshkrob nshkrob finagle: Fix tests for classpath isolation
Problem

Tests rely on how file classloaders work and fail with jar classloaders.

Solution

LoadServiceTest: read the whole buffer when loading classes.
SslTest: When resources are in a jar, there are no jar entries for directories (only files). So create a temp directory rather than extracting one from classpath.

RB_ID=647592
0492f29
@jaked jaked finagle-core: avoid allocation in Dtab.local
Problem:

Dtab.local allocates in the common case that the local Dtab is empty.

Solution:

Replace getOrElse with explicit pattern match to avoid allocation.

RB_ID=646890
52e6b2a
Dave Benjamin Isolation: Remove deprecated globs usage in master
RB_ID=647962
USER_HOOK_ARGS=--i-am-evil -x birdcage/macaw-f5/web-client/.hooks/PRESUBMIT science/src/thrift/com/twitter/ads/schema/.hooks/PRESUBMIT .hooks/PRESUBMIT/validate-config-ini science/src/thrift/com/twitter/ads/dataservice/.hooks/PRESUBMIT
81ce47b
@mosesn mosesn Revert "Revert "finagle-exp: Make BackupRequestFilter allocate less""
Problem

The old version of this review occasionally used nanoTime with a Timer, which broke it, since nanoTime should only be used for durations.

Solution

nowMs is only used for durations, never for current time.

RB_ID=649078
4426d62
Commits on May 04, 2015
Chris Chen twitter-zookeeper: remove eventually from Watcher tests.
Problem:
Watcher tests depend on eventually to assert changes in state, which can
make the test flaky if the system running the test is under heavy load.

Solution:
Set filters on conditions and use Await to allow the state to converge.

Result:
Watcher tests should be less flaky.

RB_ID=649858
3f37373
@note note quickstart doc updated
Problem

Http client snipped from quickstart does not work

Solution

Doc updated

Signed-off-by: Travis Brown <tbrown@twitter.com>

RB_ID=652608
e058eee
@atollena atollena Fix ZK session metrics doc
The documentation does not reflect the current code.

RB_ID=651974
2765af7
Commits on May 06, 2015
@blackicewei blackicewei finagle-core: fix flaky test in ClientBuilder EndToEndTest
Problem

Flaky test depends on no server is running on
a socket address to verify write exceptions.

Solution

use a fake `SocketAdress` to gaurantee write exceptions.

RB_ID=652606
79f7123
@luciferous luciferous [finagle-core] Mark ExitGuardTest flaky
Problem

Flaky test "guard creates thread, unguard kills thread".

Solution

Marking flaky.

RB_ID=654472
ebb4250
@luciferous luciferous [finagle-stream] Mark finagle-stream test as flaky
Problem

Flaky test in finagle-stream failing in CI.

Solution

Mark it flaky. Added a comment about why it sometimes fails: where we
expect the client to receive the last HTTP chunk via MessageEvent we
sometimes see an ExceptionEvent suggesting that the connection was
closed before the last chunk was sent.

RB_ID=653736
d1dd777
@roanta roanta finagle-core: Improve client Balancer docs
Problem

We don't have any documentation we can point people to that
explains the difference balancers and their motivations. I'd
like to eventually unify how we instantiate different balancers
and add code snippets for how to change / experiment with each
balancer.

Solution

Add docs to finagle/doc which is also synced to docbird and github.
Also, I added some in-code documentation as I was re-reading some
of the balancer code.

RB_ID=647240
8882e24
@kevinoliver kevinoliver finagle-metrics: Bump Commons versions
Problem

Commons Metrics is stricter than Ostrich regarding name collisions.
See RB 640376 for details on the fix that this picks up.

Solution

Bump versions of Commons to latest.

RB_ID=654708
798eeec
@dschobel dschobel finagle-httpx: cherry pick patches to sync with finagle-http
cherry pick missing patches from finagle-http onto finagle-httpx:

"introduce DynamicRecord to implement Request.ctx, Response.ctx" https://reviewboard.twitter.biz/r/570316/
???Use Netty's RFC-6585 HTTP response codes.??? https://reviewboard.twitter.biz/r/554884/

RB_ID=654306
96fc826
@stevegury stevegury Fix sbt jar retrieval
Problem

Typesafe recently moved their jar repo to bintray.
Now every jar fetching command (curl) fails because curl doesn't
follow Http 302 by default.

Solution

Add the required parameter to curl to follow the 302 redirect.

RB_ID=654736
TBR=true
624d377
Commits on May 11, 2015
@kevinoliver kevinoliver finagle: add koliver to top-level OWNERS
RB_ID=654738
5853859
Chris Chen finagle-serversets: fix Activity state leakage from ServiceDiscoverer
Problem:
ServiceDiscoverer in serverset2 leaks Activity state, triggering redundant
work and allocation in downstream computation.

Solution:
Only emit Activity changes when the state changes.

Result:
Service discovery resolution for many endpoints from the same serverset
is more efficient.

RB_ID=656822
1cc7711
@blackicewei blackicewei finagle-core: more stats around failures and draining in mux
Problem

Requeue budget is a gauge, it's hard to see when budget
exhausts due to granuality of stats collection.

There are draining/drained counters in mux.Server, but
those counters may not be collected when server shutsdown
or runs into problem.

Failures can be a string with certain flags with casue,
where flags imbed much meaning of it. It would be nice to
expose flags in stats reporting, so that we know
for example requeues or retries happen as expected.

Solution

- add a budget_exhausted counter.
- add draining/drained counters in mux.Client.
- expose Failure.flag as failure categories in StatsFilter.

RB_ID=639924
74187f8
@stevegury stevegury Goodbye
Remove stevegury from various OWNERS files.

RB_ID=658898
150c542
Evan Jones finagle-core: Implement Loaded/NullReporterFactory.toString
Problem

In the web interface, the LoadBalancer.reporter parameter has a blank
value, making it hard to understand what reporter is being used.

Solution

Implement toString with descriptive labels.

RB_ID=658992
84f7e6e
Commits on May 18, 2015
@kevinoliver kevinoliver finagle-memcached(x): Fix flakey tests
Problem

Two cache tests were marked as flakey.

Solution

Fix the tests by using `Time.withCurrentTimeFrozen`.

RB_ID=660632
0a20790
@kevinoliver kevinoliver finagle-kestrel(x): Longer Awaits for tests
Problem

These tests had been marked as flakey.

Solution

Increase the time for Await in the case of a slow
CI machine.

RB_ID=660798
294ebf7
@roanta roanta finagle: fix config for docbird
RB_ID=661948
a7ecb97
@jaked jaked finagle-serversets: slim down retained memory
Problem:

The serverset2 resolver permanently memoizes immutable data loaded from
ZooKeeper; in addition it caches some parsed representations of this
data. This takes up a lot of space.

Solution:

Permanently memoize the parsed representation only, and slim down the
representation by eliminating needless boxing and consolidating
Endpoints which differ only by endpoint name.

RB_ID=661964
ede2aff
@blackicewei blackicewei finagle-memcached: Use fixed number of connections for pipelining mem…
…cache client

Problem

When memcache client use pipelining with singleton pool,
p9999 is worse than non pipelining with multiple
connections due to head of line blocking.

Solution

Using a samll fixed number of connections, and load
balance among those connections based on outstanding
requests and time.

Result

A customized pipline memcache client, using ewma P2C
loadbalancer. With much fewer connections, Memcache
pipelining client achieve similar tail latency as non
pipelining client.

RB_ID=614228
9cd3d44
@blackicewei blackicewei finagle-core: public concurrency param for ConcurrentLoadBalancerFactory
Problem
`ConcurrentLoadBalancerFactory#Param` is private to finagle,
impossible for users to config it.

Solution
Change it to be public

RB_ID=665390
12336c5
Commits on May 21, 2015
@roanta roanta finagle-core: unify loadbalancer constructors
Problem

Constructing a new Balancer that can be injected into a Finagle client
is unnecessarily complex and non-uniform.

Solution

Remove the legacy constructors around defining the collection of endpoints
and simplified the interface to `LoadBalancerFactory`. Now,
`com.twitter.finagle.loadbalancer.Balancers` defines the collection of balancer
constructors which are uniform and also compatible from Java.

RB_ID=660730
TBR=true
USER_HOOK_ARGS=--i-am-evil -x birdcage/macaw-swift/.hooks/PRESUBMIT/check-swoop
8d056de
@dschobel dschobel birdcage: rm deprecated kestrel package and s/kestrelx/kestrel/
There are no more users of finagle-kestrel in Source so we can rm it.

Patch created with:

git rm -r finagle/finagle-kestrel/
git mv finagle/finagle-kestrelx/ finagle/finagle-kestrel/
perl to s/kestrelx/kestrel/ + some manual deletion of duplicate kestrel refs

RB_ID=667920
TBR=true
32b30da
@roanta roanta finagle-core: Remove Aperture flags
Problem

Configuring per-client settings globally is generally not a good idea.

Solution

Remove the aperture flags and update our internal usage to configure it
per-client configuration instead.

RB_ID=663194
TBR=true
USER_HOOK_ARGS=--i-am-evil -x birdcage/macaw-swift/.hooks/PRESUBMIT/check-swoop
5c0250d
@kevinoliver kevinoliver finagle: Add registry entry for latched counters
Problem

Its not immediately obvious if counters are latched or not.

Solution

Introduce an Registry entry that exposes this.

Result

A snippet from a service's /admin/registry.json
now gives you this information clearly.

    "stats" : {
      "commons_metrics" : {
        "counters_latched" : "true"
      }
    },

RB_ID=667890
841e333
ddavydov Fix target types of 'fake' jar library targets
RB_ID=668822
TBR=true
NO_USER_HOOK=1
04db343
@dschobel dschobel finagle-doc: de-nettyify quickstart
Use httpx for quickstart docs. Bonus version bump to latest stable packages.

RB_ID=669490
6792489
@kevinoliver kevinoliver finagle-core: Clarify synchronization in WatermarkPool
Problem

It was not very obvious to the reader if the synchronization
in WatermarkPool was always synchronizing on the WatermarkPool.

Solution

Alias WatermarkPool to thePool and use that for synchronization
consistently.

RB_ID=670634
a218785
Commits on Jun 01, 2015
Dan Richelson publish finagle-memcached(x) internally
RB_ID=673810
23ad89c
@kevinoliver kevinoliver finagle-exception,zipkin: Improve client configuration
Problem

The Finagle clients used by Reporter and RawZipkinTracer are not well
behaved in the face of a poorly behaved server and there is no
visibility into their stats.

Solution

Export stats to ClientStatsReceiver, add an upper bound on max waiters
and a global timeout of 1 second.

Result

Visibility into the client behavior and removal of an unbounded queue
which can manifest as a memory leak.

RB_ID=674628
eb94564
@whiter4bbit whiter4bbit update written bytes statistics after completion
Signed-off-by: Bing Wei <bing@twitter.com>

RB_ID=675592
65eb49d
@kevinoliver kevinoliver finagle-exception: ExceptionReporter is no longer the default service
Problem

finagle-exception declared ExceptionReporter as the default service
for `com.twitter.finagle.util.ReporterFactory`.

Solution

Remove the declaration from META-INF/services

RB_ID=674646
bd67b1b
Dan Knightly Add Spnego Mechanism to SpnegoAuthentication filter.
 Problem:  The SpengoAuthentication filter rejects Authentication using the Spnego mechanism.  The filter is currently setup to only allow the Krb5 mechanism.

 Solution:  Configure the Server GSSContext to accept both Krb5 and Spnego mechanisms.

 Result:  The Filter authenticates requests using either krb5 or Spnego mechanism and valid credentials.

RB_ID=677218
0f8f733
@blackicewei blackicewei finagle-httpx: nack support
Problem

When HTTP server throws retryable exceptions, it fails
the request and the connection is torn down.

Solution

Introduce a new HTTP header "finagle-http-nack". When server
fails with retrable exceptions, it sends back Nack response,
i.e. 503 response with the header. Client auto reenqueues
those requests.

RB_ID=670046
e7d2a8c
@tw-ngreen tw-ngreen finagle-core: Add Annotations indicating ServiceFactoryCache failure
Problem

If the ServiceFactoryCache, or in fact anything else in the request
filter, fails, the zipkin trace shows just two annotations:
ClientSend
ClientReceive

This is deceptive as there was nothing sent or received, and there was
an error that is not indicated here.

Solution

1. Annotate cache misses when the ServiceFactoryCache begins naming,
and annotate with success on success and failure on failures including
when service acquisition times out.

2. Add an event for when the request is actually sent over the wire,
since "Client Send" does not in fact indicate that the client has sent
anything.

3. If there is an exception in the request chain, annotate the exception
in addition to Client Receive so it's clear that something has gone
wrong.

Result

All successful client spans now have 4, not 2, events:
ClientSend
WireSend
WireReceive
ClientReceive

Traces with ServiceFactoryCache errors (and other errors) now add
annotations on exceptions.

RB_ID=653628
75e8c18
Commits on Jun 08, 2015
@tnarg tnarg lleweraf
Problem
tnarg is leaving

Solution
Remove tnarg from OWNERS files

Result
No one else has to do this

RB_ID=679802
cbf6c69
@blackicewei blackicewei finagle: finalize isAvailable in Service and ServiceFactory
Problem

In `Service` and `ServiceFactory`, `Status` supresses
`isAvaiable`. There is no need to override `isAvaible`
in subclasses.

Solution

Finalize `isAvaiable` method.

RB_ID=678588
a0d5ee7
@roanta roanta finagle-benchmark/util-benchmark: Bump jmh version
RB_ID=679950
1e81a0a
@kevinoliver kevinoliver finagle-core: Use LongAdder in StatsFilter
Problem

StatsFilter uses an AtomicInteger where an LongAdder might give
better performance under contention.

Solution

LongAdders should give better update performance at the tradeoff of
more memory used.

RB_ID=678512
0018434
Chris Chen ZooKeeper testing: Remove references to BasicDataTreeBuilder
Problem:
In ZOOKEEPER-1200, BasicDataTreeBuilder was removed from ZooKeeperServer. This
breaks existing tests that instantiate a ZooKeeperServer referencing the
builder.

Solution:
Switch instantiation to a different constructor that takes log path names
and a tick time.
Coordination team takes ownership of common/common_internal zookeeper.

Result:
We can upgrade the zookeeeper artifact in source to 3.5+.

RB_ID=678658
bc97852
@blackicewei blackicewei finagle-core: propagate Status in ServiceFactory.flatMap
Problem

`Status` is not override correctly in `ServiceFactory.flatMap`.
It defaults to `Status.Open`, which drops the failure detection
results from other layers of the stack. It results in
unexpected behavior in admimission controllers or requeues which
depends on the `Status` value.

Solution

Override `Status` in ``ServiceFactory.flatMap`.

RB_ID=677986
2f1343a
@kevinoliver kevinoliver finagle-memcachedx: Client performance improvements
Problem

The memcached client could be more performant.

Solution

Made many small changes that will help in aggregate:

AbstractDecoder:

* `decodeLine`s needsData now uses an Int and a sentinel instead of
  `Option`

Decoder:

* Avoid allocating a closure for AwaitingResponse's call to decodeLine

* Use :+ instead of wrapping a single element in a Seq and using ++

* Converted `AwaitingResponse` to a singleton case object

* Use a singleton `EmptyValueLines` when possible

RichChannelBuffer:

* Is now a value class to avoid unnecessary allocations

* Optimized `toInt` to avoid conversion to String

* Properly sized the `ArrayBuffer` in `split`

* Replace `scratch.slice(0, 0)` with `ChannelBuffers.EMPTY_BUFFER`

ConnectedClient:

* Use `collection.breakOut` in `rawGet`

* Use `Future.flatMap` and constants for `Future(true|false)`

RB_ID=680348
6486790
Commits on Jun 12, 2015
@kevinoliver kevinoliver finagle-stats: Give details when perHostStats are disabled
Problem

It is not obvious on /admin/per_host_metrics.json if per host stats
are being collected or not.

Solution

Detect if the perHostStats flag is disabled and provide a response
guiding them on how to enable the feature.

RB_ID=687636
cb81506
@kevinoliver kevinoliver finagle-stats: Add explicit dependencies
Problem

finagle-stats directly uses some code that is not explicitly
declared in its BUILD file.

Solution

Add the explicit dependencies.

RB_ID=688276
518d675
@blackicewei blackicewei finagle-thriftmux: avoid convert restartable exceptions to TApplicati…
…onEx

Problem

`c.t.f.thriftmux.ThriftMux.Server.ExnHandler` in the
ThriftMux stack converts any non-`org.apache.thrift.TException`
to `org.apache.thrift.TApplication`, which is encoded as
a successful mux response (RdispatchOk). Thus, restartable
failures from service level are not reenqueued on the client side.

Solution

Does not convert restartable exceptions in ExnHandler.

Result

When thriftmux server talks to thriftmux client, restartable
failures are returned as RdispatchNack response, thus auto
retried on client side.

RB_ID=686582
14b631c
@kevinoliver kevinoliver finagle-core: Fix ExitGuardTest
Problem

ExitGuardTest assumes that no other ExitGuards have been registered
and remain guarded.
This is not the case when running finagle-core's tests.

Solution

Fix the test to handle both categories.

RB_ID=690240
b30fd48
@kevinoliver kevinoliver finagle-core: Remove allocations from HandletimeFilter
Problem

HandletimeFilter uses Stopwatch when it does not need to.

Solution

Use `System.nanoTime` and compute elapsed time directly.

RB_ID=689140
0eac1ad
@luciferous luciferous [finagle-http] Preserve reader and writer state when constructing req…
…uest and response

Problem

The constructor for Responses and Requests that takes HttpResponse and
HttpRequest should perform a match to see if the parameter is already a
Response or Request, and in those casesjust return them. This is so that
the state of thereader and writer is preserved.

Solution

Return the param if it's already the correct type, and if we need to
create a new object, make sure to copy the reader and writer.

RB_ID=690768
6359210
@luciferous luciferous [finagle-core] Netty3 corralling
Problem

Corralling Netty 3 handlers into their own namespace. This will help
isolation when we introduce Netty 4 corresponding types into the space.

Solution

Move types that will have Netty 4 specific implementations into netty3
namespace (the Netty 4 types will presumably be under netty4).

RB_ID=691746
5e13b9e
@nshkrob nshkrob finagle-thrift: Document the buffer/resetCount stat
Problem

Missing docs for thrift/buffer/resetCount stat.

Solution

Add docs;

Also fix the duplicate /thrift in the stat name.

RB_ID=691388
146081e
Commits on Jun 15, 2015
@roanta roanta finagle-core: Change default balancer to P2C
Problem

The P2C balancer is strictly better than the heap based balancer
for our intended uses. However, the heap is the default for
Finagle clients.

Solution

Change the default load balancer for Finagle clients to P2C.
To override this, use the `com.twitter.finagle.loadbalancer.Balancers`
API per finagle client.

```
val balancer = Balancers.heap()
Protocol.configured(LoadBalancerFactory.Param(balancer))
```

RB_ID=693450
7cecfec
Commits on Jun 22, 2015
@luciferous luciferous [finagle] Remove finagle-memcached
Problem

finagle-memcached is deprecated in favor of finagle-memacachedx.

Solution

Remove finagle-memcached.

RB_ID=694580
85cee04
@nshkrob nshkrob finagle: Fix flaky BalancerStackModuleTest
Problem

BalancerStackModuleTest has a race between Addr resolution and the metric being available.

Solution

Use Eventiually.

RB_ID=695440
aab64af
@roanta roanta finagle-docs: Update client docs wrt default balancer
RB_ID=694830
c0522ee
@kevinoliver kevinoliver Add RB_IDs to CHANGES
Problem

It should be easier to see the details of what motivated
a given change in our CHANGES files.

Solution

Add the ids to the changelogs.

RB_ID=695076
f60073b
@WamBamBoozle WamBamBoozle culling stale owners
Removed 151 former employees from 704 files. Removed 74 empty files.

RB_ID=695668
TBR=true
5daac70
Eoin Coffey finagle/mysql Client.transaction correctly resolves after ROLLBACK
 Problem: finagle/mysql Client.transaction can fail the returned future
 before finishing the transaction rollback

 If the transaction block fails, then we need to `"ROLLBACK"` the DB
 transaction. Previously we did this via `respond`, which meant that the
 Future transaction returns could resolve before the DB transaction is
 rolled back. This is confusing and surprising to client code.

 Solution: Use `transform` instead of `respond` for mysql transactions.

 Instead we use `transform` and chain the resolving the future to the
 resolution of the `"ROLLBACK"`.

RB_ID=695984
fe4057b
@kevinoliver kevinoliver finagle-core: Fix loading of build.properties
Problem

The finagle version was not always being read properly
in `com.twitter.finagle.Init` leading to an unknown
version used for `Init.finagleVersion`.

Solution

Look in more locations for the version and add a test
to verify that this does not regress in the future.

RB_ID=694814
efce5ec
@kevinoliver kevinoliver finagle-core: Customizable TimeUnit for StatsFilter
Problem

For some use cases the millisecond timeunit resolution of
`com.twitter.finagle.service.StatsFilter` is too coarse grained.

Solution

Allow the TimeUnit to be customizable.

RB_ID=695258
88193d8
@kevinoliver kevinoliver finagle: Make BackupRequestFilter less resource intensive
Problem

BackupRequestFilter is resource intensive in terms of both resident
memory size as well as CPU.

Solution

Changes to use less resident memory as well as lessen the cost of
sending a backup request.

BackupRequestFilter

 * Only recalculate the cutoff for a percentage of requests.

 * Minimize allocations and unnecessary work when sending a backup
   request.

LatencyHistogram

 * Introduce `error` which controls the granularity of how accurate
   the latency quantile will be. This allows you to tradeoff resident
   memory for accuracy.

WindowedAdder

 * Examined switching to `AtomicLongs` from `LongAdders` but didn't
   make the switch. If there is no contention, there is little extra
   in terms of memory used. However, we may at some point decide the
   minimal performance wins even under heavy contention are not worth
   the memory footprint overhead.

Result

JMH results. Note, error=-1.0 disables errors and perfoms as the baseline.

[info] Benchmark                       (error)  (maxDurationMs)  Mode  Cnt      Score      Error  Units
[info] LatencyHistogramBench.add          -1.0             1000  avgt   10    257.410 ±   27.619  ns/op
[info] LatencyHistogramBench.add          0.01             1000  avgt   10    256.716 ±   47.596  ns/op
[info] LatencyHistogramBench.add          0.05             1000  avgt   10    224.338 ±   30.683  ns/op
[info] LatencyHistogramBench.quantile     -1.0             1000  avgt   10  60711.140 ± 6873.934  ns/op
[info] LatencyHistogramBench.quantile     0.01             1000  avgt   10   7116.885 ±  813.226  ns/op
[info] LatencyHistogramBench.quantile     0.05             1000  avgt   10   1499.244 ±  322.476  ns/op

Minor win on the order of 50 bytes/op in terms of memory allocations
for `BackupRequestFilter.apply`.

By recalculating the cutoff for only 1% of requests, this saves 59 µs
for the other 99% of requests.

RB_ID=693552
20acce3
@nshkrob nshkrob finagle: Disable MdnsTest in CI
Problem

MdnsTest announcer and resolver communicate over network, which doesn't work in CI.

Solution

Disable the test in CI by marking it flaky.

RB_ID=697028
ee88357
@blackicewei blackicewei finagle: add stats to failureDetector
Problem

FailureDetector expose minimal stats, making it hard to debug.

Solution

add counters and stat.

RB_ID=697128
d00e7bb
@roanta roanta csl/integ: Introduce a balancer test into our integration suite.
RB_ID=698056
abb4a4c
@vkostyukov vkostyukov finagle: Refactor FailureAccrual Stack Params
RB_ID=689076
c89d5a9
@jamescway jamescway make trace stack thread local
#363

Problem

Trace's global stack variable is not thread safe.

Solution

Changing to be thread local for thread safety.

Result

"stack" acts as a thread local variable and cannot be
shared between threads.

Signed-off-by: Nik Shkrob <nshkrob@twitter.com>

RB_ID=699752
086f362
Todd Segal finagle-mux: Make FailureDetection parameters programmatically config…
…urable and stackable

Problem:

FailureDetection parameters are currently only settable via global flags. Apps should be able to configure these programatically either directly or through stackable parameters.

Solution:

Create a FailureDetector companion object which can instantiate a failure detector based on config. Make the param embeddable and extractable from stack params for use.

RB_ID=698704
e5dff7b
Commits on Jun 23, 2015
@adleong adleong Filter.TypeAgnostic
TypeAgnostic filters are like SimpleFilters but they but they leave the Rep and Req types unspecified until `.toFilter` is called.  This allows a TypeAgnostic filters to be applied to services with different type parameters.

RB_ID=675046
2ca3721
@blackicewei blackicewei finagle-memcachedx: enable FactoryToService in pipelining client
Problem

Memcached pipeling client does not have FactoryToService
enabled, so it retries service acquistion up to 25 times
in `Requeues` module, this causes connection retry storm
when memcache applies connection rate limit on the server
side.

Solution

Use `StackClient.newService` to build the client, so the
leaky bucket retry logic kicks in during service acquistion.
That gives us an upper limit on number of retries based
on qps.

RB_ID=700568
0b9cb36
@atollena atollena [finagle-core] Prefer Ok when binding NameTree Unions
Problem
--

We use Activity.collect to aggregate NameTree unions on binding.  As a
consequence, the resulting Activity is Failed (or Pending) if any of the
union members is Failed (or Pending).

A typical use case for weighted unions is failover:

    /s => 75 * /srv/dc1 & 25 * /srv/dc2

If either one of `/srv/dc1` or `/srv/dc2` is pending resolution, the
entire result is pending regarless of the state of individual
components. As a result 100% of requests fail.

Solution
--

Implement custom rules for aggregating NameTree union activities:

- if one or more components of the union are in Ok state, return them
- if all components are in pending state, the result is pending
- if no member is in Ok state, and some are Failed, return the first
  failure (this matches Activity.collect behavior).

Result
--

We can safely use NameTree unions for cases where one component may fail
or be left pending.

RB_ID=697114
88a862f
@luciferous luciferous [finagle-stream] Make dispatchers less private
Problem

Stream dispatchers aren't visible to stream users who want to customize
the StreamCodec.

Solution

Increase visibility slightly.

RB_ID=701952
ae65b7d
@dschobel dschobel release CSL OSS
release new minor version of all CSL OSS.

twitter-server: 1.11.0
scrooge: 3.19.0
finagle-*: 6.26.0
ostrich: 9.9.0
util: 6.25.0

RB_ID=701418
TBR=true
f8ea987
Commits on Jun 29, 2015
@olix0r olix0r finagle-core: Improve Name.Bound documentation.
Problem

The 'id' and 'path' fields of Name.Bound were insufficiently documented.

Solution

Clarify that 'path' should only contain unbound residual path components
and that 'id' should only reflect the bound path.

Github PR #378

Signed-off-by: Daniel Schobel <dschobel@twitter.com>

RB_ID=704664
de6c058
@atollena atollena [finagle-core] trivial style fixes to Namer
(followup on RB 697114)

RB_ID=705450
480e4e3
@atollena atollena [finagle-core] Introduce NameInterpreter
Problem
--

The Dtab and the name resolution mechanism are conflated: it is not
possible to resolve a name without first resorting to the Dtab, and the
Namer interface does not take the Dtab into consideration.  Resolution
must happen partly in the client.  This prevents outsourcing name
resolution and centralizing the Dtab into a separate service in a way
that retains existing use-cases.

Solution
--

Add an optional indirection, NameInterpreter, between (dtab, path) and
bound names, used by BindingFactory.

Result
--

We can override how dtab and path interact.

RB_ID=689726
e26e033
Commits on Jul 06, 2015
@blackicewei blackicewei finagle-http: support Nack
add Nack support in http as well for users haven't upgraded to httpx

RB_ID=705948
7a9f14f
@travisbrown travisbrown Fix Travis CI and add code coverage for Finagle and related projects
Problem

This change addresses three closely related problems. The first is that our
Travis CI builds have been broken for a long time, in part because of the fact
that since we don't publish snapshots, the current development versions aren't
able to find their dependencies unless these are published locally. (Travis CI
also recently changed containerized builds to be the default, and we're still
unable to use the containerized build system with Finagle because of travis-ci
issue #3120, so I've turned this off for the relevant projects.) The second
issue is that we're not currently clearly indicating project status (whether a
project is retired, deprecated, or actively maintained). Finally, we want to
publish code coverage information for these projects.

Solution

This commit extends the work in Scrooge to provide a Travis CI script that
downloads and publishes dependencies locally. It also adds retries for the
dependency resolution and downloading step for each Travis CI config, as well as
caching for dependencies. It also adds a shields.io badge to the top of each
project README indicating the status of the project, and turns on Scoverage code
coverage reporting (and publication to Coveralls) for each project.

Result

If our Travis builds are failing, it's much more likely to be for legitimate
reasons. We also now clearly indicate code coverage stats and project status.

RB_ID=704444
e42d011
@yschimke yschimke finagle-httpx: Adds common mimetypes
Problem

MediaType doesn't define constants for image types.

Solution

Add extra constants to MediaType.

RB_ID=700876
1664d98
@luciferous luciferous finagle-stream: Remove netty types from public API
Problem

finagle-stream leaks netty types in its public API, this should be a
hidden implementation detail.

Solution

Replace netty types with more portable types.

RB_ID=695896
2bf91d7
@luciferous luciferous [finagle-example] Remove stream examples
Problem

finagle-http/x has a good story for HTTP streaming, finagle-stream is
redundant and we want to discourage further adoption of it.

Solution

Remove stream examples.

RB_ID=708944
12b2f5f
@luciferous luciferous [finagle-example] Reinstate streaming examples using httpx
Problem

The streaming examples that were removed are actually quite interesting.
There is an HTTP streaming story, we should use that instead of
ommitting streaming examples entirely.

Solution

Reinstate the streaming examples, but using finagle-httpx instead of
finagle-stream.

RB_ID=709946
afd6d14
@luciferous luciferous [finagle-httpx] Java friendly Method and Version
Problem

Method and Version are difficult to use from Java (e.g.
Method.Get$.MODULE$).

Solution

Provide Methods and Versions for Java usage.

RB_ID=710187
6ec9d2e
@vkostyukov vkostyukov finagle|util: Mention Java compilation tests in CONTRIBUTING.md
RB_ID=710340
074bf97
@luciferous luciferous [finagle-stream] Stream dispatcher not converting headers
Problem

The server dispatcher is missing the headers in the conversion from
StreamResponse to the underlying codec type (HttpResponse).

Solution

Fix the StreamResponse conversion so that everything is properly
converted. Add a test.

RB_ID=710513
0557ccd
@nshkrob nshkrob finagle-thrift: Replace manually-compiled files with a thrift target
Problem

finagle/finagle-thrift/src/main/java/com/twitter/finagle/thrift/thrift/ has java files that can be generated from tracing.thrift instead.

Solution

Replace the manually-generated files with the proper thrift target. This was blocked by lack of java support in scrooge-sbt-plugin.

RB_ID=707520
26dae34
Commits on Jul 13, 2015
@atollena atollena [finagle-core] Counter measuring initial resolution time
Problem
--

Initial client resolution takes a long time, but we don't know
how long.  We want to track how much we gain on startup by using
a different service discovery mechanism.

Solution
--

Add a counter.

RB_ID=710201
e9c472b
@cacoco cacoco finagle-core: Close resource in Init and expose finagleBuildRevision
Problem

com.twitter.finagle.Init attempts to load and read the
/com/twitter/{finagle-core}/build.properties file currently by opening an
InputStream to the resource but never closing the stream. Additionally, there
is the need to consume the "build_revision" value from the build.properties.

Solution

Rewrite the tryProps(..) method to ensure that the opened InpuStream is closed
after reading the Stream into a java.util.Properties object. Also, expose a
new method: "finagleBuildRevision" which returns the value of the
"build_revision" key in the found Finagle build.properties.

Result

The open file descriptor for the loaded resource will be properly closed and
consumers will access to the Finagle "build_revision" value through the
com.twitter.finagle.Init#finagleBuildRevision(..) method.

RB_ID=711518
4d03fa4
@kevinoliver kevinoliver finagle-core: Improve performance of Filter.andThen.apply()
Problem

c.t.finagle.Filter.andThen's `apply()` method is a very commonly used
method in Finagle services.

Solution

Copy-paste a bit of code to avoid allocations and improve performance.

Result

Less allocations and better performance.

Baseline:
[info] Benchmark                                                       (numAndThens)  Mode  Cnt     Score      Error   Units
[info] FilterBenchmark.andThenFilter                                               1  avgt   10    14.629 ±    7.833   ns/op
[info] FilterBenchmark.andThenFilter:·gc.alloc.rate.norm                           1  avgt   10    48.000 ±    0.001    B/op
[info] FilterBenchmark.andThenFilter                                              10  avgt   10   185.234 ±  121.924   ns/op
[info] FilterBenchmark.andThenFilter:·gc.alloc.rate.norm                          10  avgt   10   504.000 ±    0.002    B/op
[info] FilterBenchmark.andThenFilter                                              20  avgt   10   253.850 ±   23.857   ns/op
[info] FilterBenchmark.andThenFilter:·gc.alloc.rate.norm                          20  avgt   10   984.000 ±    0.002    B/op

Patched:
[info] Benchmark                                                       (numAndThens)  Mode  Cnt     Score     Error   Units
[info] FilterBenchmark.andThenFilter                                               1  avgt   10     8.644 ±   0.441   ns/op
[info] FilterBenchmark.andThenFilter:·gc.alloc.rate.norm                           1  avgt   10    24.000 ±   0.001    B/op
[info] FilterBenchmark.andThenFilter                                              10  avgt   10   103.514 ±  35.700   ns/op
[info] FilterBenchmark.andThenFilter:·gc.alloc.rate.norm                          10  avgt   10   264.000 ±   0.001    B/op
[info] FilterBenchmark.andThenFilter                                              20  avgt   10   145.070 ±   4.226   ns/op
[info] FilterBenchmark.andThenFilter:·gc.alloc.rate.norm                          20  avgt   10   504.000 ±   0.001    B/op

RB_ID=711037
44e62e8
@vkostyukov vkostyukov docs: Add the "why-migrate" section to the util-stats docs
RB_ID=709505
6f7c6a8
@legalosLOTR legalosLOTR finagle-core: Support passing http proxy credentials to ClientBuilder
Problem
- ClientBuilder allows you to set a httpProxy but you cannot specify credentials for auth, therefore it cannot work with authenticated http proxies (like the one we use with mesos).

Solution
- Add method to pass in credentials to ClientBuilder.
- If credentials are supplied, add a Proxy-Authorization header to the CONNECT request.

RB_ID=710481
526aef0
@luciferous luciferous finagle-httpx: Set Content-Length header
Problem

The client dispatcher is responsible for setting the Content-Length
header on outgoing requests, but it doesn't.

Solution

Set the Content-Length header.

RB_ID=713124
e2f4187
@blackicewei blackicewei finagle-mux: unmap promise from tagMap to avoid races
Problem

There is a race where promises in mux dispatcher can be
updated twice.

Solution

`unmap` the `promise` from `TagMap` before setting a
value to the `promise`. `TagMap`'s synchronized operations
gaurantee a `promise` won't be fetched twice, thus be set
a value twice.

RB_ID=712013
a10d17e
@dschobel dschobel finagle-http|x: add dschobel to owners
RB_ID=713162
158250c
@atollena atollena [finagle-core] Remove Namer trait from Dtab
Problem
--

We have multiple ways to bind a name via a Dtab: Dtab itself
is a `Namer`, and `NameInterpreter` convert a Path to a bound
name, taking the Dtab into account.

Solution
--

Make local dtab resolution a NameInterpreter.

Result
--

One mechanism for resolving a path to a bound name using the
dtab.

RB_ID=711681
TBR=true
719d505
@atollena atollena [finagle-core] Properly recurse to the Dtab when namers return an unb…
…ound name

Problem
--

When looking up a name via the DefaultInterpreter, if the Dtab lookup
yields Neg we directly bind via global namer (which resolves names of
the for /$), and never recurse back to the Dtab. This breaks dtabs
refering to namers that return an unbound names.

Solution
--

Properly use Namer.global.lookup rather than Namer.global.bind.

RB_ID=713828
c4440f1
@atollena atollena [finagle-core] Add a test for namers that return unbound names
Problem
--

No test caught the fact that DefaultInterpreter didn't call back to the dtab
when namers return unbound names.

Solution
--

Add a test.

RB_ID=713856
53aaa8f
@dschobel dschobel finagle-http: set content-length header in netty http client dispatcher
Problem

The client dispatcher for netty http doesn't define the content-length
header for non-chunked requests.

Solution

Set the header.

RB_ID=714267
d9276f9
@travisbrown travisbrown Add missing SBT dependency
RB_ID=714548
3bbb502
Commits on Jul 20, 2015
@kevinoliver kevinoliver finagle-zipkin: Improve concurrency of DeadlineSpanMap
Problem

`c.t.f.zipkin.thrift.DeadlineSpanMap` uses coarse grained locks for
thread-safety.

Solution

Use finer grained synchronization on the individual MutableSpan's via
a `j.u.c.ConcurrentHashMap`.

RB_ID=714446
ed7e492
@blackicewei blackicewei finagle-memcachedx: reduce compile warnings
reduce compile warnings

RB_ID=714976
72261ed
Tejal Desai [MonoBuild] Remove exclude from birdcage/finatra and move it to deped…
…encies.

During monobuild verification with source deps,  org.slf4j.slf4j-log4j12 is missing from few bundles of science targets. This is due excluded added in finatra/finatra-serversets in this rb  https://reviewboard.twitter.biz/r/641898.

Moving this exclude to the dependees in birdcage.

RB_ID=715436
d3f4537
@blackicewei blackicewei finagle-core: exclude UnresolvedAddressException from RetryableWriteE…
…xception

Problem

All `WriteException`s are retryable, and all Netty level IO
exceptions are wrapped as `WriteException`. That's the case
for `UnresolvedAddressException`, which should not be retried.

Solution

Exclude `UnresolvedAddressException` from `RetryableWriteException`.

RB_ID=714007
c3dd3cb
Jillian Crossley finagle-core: Made Client/Server Tracing Filters not private[finagle]
Problem

The client and server tracing annotations are required and used to live in the codec tracing filter (replaced with TraceInitializerFilter, etc.).

Solution

Made (Client|Server)TracingFilter public.

RB_ID=714265
500ff96
@stuhood stuhood Disable flaky finagle-stats test
RB_ID=716108
TBR=true
8b1884f
@vkostyukov vkostyukov twitter-server|finagle: Enable syntax highlighting in the docs
RB_ID=710457
2c20142
@kevinoliver kevinoliver finagle-core: Expose Netty's TrafficClass configuration
Problem

There is no way for clients or servers to configure Netty's traffic
class[1].

Solution

Expose this via a new Stack.Params `Listener.TrafficClass` and
`Transporter.TrafficClass`.

[1] http://netty.io/3.10/api/org/jboss/netty/channel/socket/SocketChannelConfig.html#setTrafficClass(int)

RB_ID=714721
0c2222f
@luciferous luciferous finagle-core: Label FailFastExceptions for more useful log messages
Problem

It's difficult to know which endpoint is marked dead from the log
message of a FailFastException. While this can be inferred by observing
"marked_dead" counter increments in the same time range, adding the name
of the endpoint to the exception message makes it much easier.

Solution

Customize the exception message with a Label in the Stack scope.

RB_ID=716021
9ac7f79
@nshkrob nshkrob util: Don't forward interrupts to a completed Future
Problem

Promise.forwardInterruptsTo(other: Future) allocates chains of Transforming state objects. This is unnecessary if the other future is completed.

Solution

Make forwardInterruptsTo a no-op if the other future is completed.

RB_ID=714420
8c4420f
@kevinoliver kevinoliver util, finagle, twitter-server: Introduce Sink.recording
Problem

Calling into `Sink.event` can be too expensive, even when using a
`NullSink`.

This is due to the expense of finding the Trace Id and making all of
the other calls.

Solution

Introduce the notion of recording which can be toggled at runtime via
/admin/events/{recordOn,recordOff} and gets a nice UI treatment of
/admin/events.

RB_ID=715712
f7c031d
Tejal Desai Remove excludes from few targets for monobuild
 Remove excludes for monobuild and add it dependees.
 This is a follow up from https://reviewboard.twitter.biz/r/715436/

RB_ID=716431
TBR=true
fecf832
@cacoco cacoco site: Update Finatra project link and blurb
     Problem

     The finagle/site/index.html has an outdated github link to the Finatra project.

     Solution

     Updated the link to the official github project and update the blurb.

     Result

     The rendered page in github will point users to the correct place for the Finatra project.

RB_ID=717011
7faddd5
@nepthar nepthar [finagle core/mux] - Delay service creation until ssl handshake compl…
…etes, add ssl peer cert to ThriftMux

Problem

When attempting to use 2-way TLS implemented earlier[1], two issues were encountered:
- Service-related objects are created before the SSL handshake completes, preventing capture of peer certificate information
- ThriftMux, which is widely used, has its own dispatcher and thus peer certificate information is not propagated

Solution

- Delay the creation of service-related objects until after the SSL handshake completes. This is done by having the channelConnected event wait on the TLS handshake
- Combine the SSL/TLS-related handling code and rename appropriately
- Add the appropriate Context to ThriftMux's dispatcher

Result

Peer certificates are now available to services using both standard and mux thrift.

[1] https://reviewboard.twitter.biz/r/631870/

RB_ID=681872
e0ebcfa
Commits on Jul 27, 2015
Chris Chen Twitter-zookeeper: Make integration tests ready for ZK 3.5+
Problem:
Existing Apache ZK tests are locked to the API as it existed with ZK 3.4.

Solution:
Update test usage to use APIs available in 3.4 and 3.5+.

Result:
Tests works with 3.4 and 3.5+.

Note: Preallocation calls in ZKServerWrapper were added by tnarg@, who
said he added them in an attempt to see if there was a breaking point
in the server code. Reverting that change to keep things clean.

RB_ID=716700
5ad2f4e
@vkostyukov vkostyukov finagle: Kill RichChannelFuture
RB_ID=717720
038dc3f
@ity ity Switch Source to Monobuild - go/monobuildproject
RB_ID=709801
NO-QUEUE=true
ecbe74d
@svetlyak40wt svetlyak40wt finagle: Fix 6.26.0 header level in CHANGES
Signed-off-by: Vladimir Kostyukov <vkostyukov@twitter.com>

RB_ID=717983
d1d0c41
@luciferous luciferous Introduce ServiceFactorySocketAddress so that Names may bind local se…
…rvices

Problem:

When building a client using a Name--i.e. via Dtabs--there is no mechanism to
cause a client to bind to a local Service reference.

Imagine, for example, a program `zoo` that exposes exposes an http
endpoint `/api/birds`; and satisfying these requests requires a
downstream client to a service `/s/birds`.

We may typically run `zoo` with a base dtab like:

    /s => /$/io.bouyant.sd

`/s/birds` may be served by a downstream service, but it may also be
served from directly within the process (e.g. by an in-memory buffer).
This situation may arise when migrating functionality between processes
(e.g. decomposing a service) or mocking downstream clients. In these
cases, we want these requests routed in-process with a dentry like:

    /s/birds => /$/io.buoyant.birds.local

The workaround today is to have zoo bind to an ephemeral port and
control redirects to that ephemeral port:

    /localPort => /$/inet/127.1
    /s/birds => /localPort$/${birdsPort}

However, this approach incurs a full serialization roundtrip.

Solution:

Introduce a SocketAddress type that encapsulates a ServiceFactory.
StdStackClient's default endpointer bypasses the client's Transporter
when such an address is encountered. In the example described above, we
bypass the entire networking stack for local requests.

Result:

Local services may be named.

Signed-off-by: Neuman Vong <nvong@twitter.com>
Github: Closes #396

RB_ID=717989
bc9b0eb
@spockz spockz Changed localhost to 127.0.0.1 to prevent ipv6 ip
Problem

Tests fail in environments that have both IPv4 and IPv6 interfaces.
This is because the endpoints are added to the load-balancer twice and
therefore the tests fail that expect to observe only a single message.

Solution

Change localhost to 127.0.0.1 in the affected tests.

Signed-off-by: Neuman Vong <nvong@twitter.com>
Github: Closes #353

RB_ID=718181
c849035
@vkostyukov vkostyukov finagle-stats: Allow to pass separator (as flag) to MetricsStatsReceiver
RB_ID=717129
9c76b6a
@vkostyukov vkostyukov finagle: Add query benchmark for CommonsStatsReceiver (upgrade JMH)
RB_ID=717603
de232ad
@kevinoliver kevinoliver finagle-stats: Improve perf of Stat.add
Problem

The cost of finagle-stats' `Stat.add()` is too expensive.

Solution

Various changes that drastically improve the performance:

* The separate tracking of average/min/max is now part of
  BucketedHistogram. This drops support for standard deviation
  which is too expensive to track.

* Member variables in BucketedHistogram are `private[this]`.

* MetricsBucketedHistogram's windowing is faster, simpler and
  more space efficient. It uses 2 windows, current and previous
  while the previous implementation used 4: 3 for previous plus
  1 current.

Result

The performance is now more or less bound to the performance
of `Arrays.binarySearch`.

RB_ID=717647
5f9426c
@thirstycrow thirstycrow finagle-core: LoadService may fail with StackOverflowError
Signed-off-by: Vladimir Kostyukov <vkostyukov@twitter.com>

RB_ID=718937
206eb46
@roanta roanta finagle-mysql: Add missing parameter case to avoid NPE in Parameter#u…
…nsafeWrap

Problem

Parameter.unsafeWrap can throw an NPE if the unknown parameter
is null.

Solution

Special case null to Parameter.NullParameter.

RB_ID=718399
b14f232
@nshkrob nshkrob scrooge: Generate a finagle Service per thrift method
Problem

Scrooge-generated services are not finagle Services, so they do not allow using Filters.

Solution

Generate a wrapper service that delegates to the underlying thrift service.

Result

For a thrift service, e.g.

service Logger {
  string log(1: string message, 2: i32 logLevel);
  i32 getLogSize();
}

Scrooge generates:

class Logger$ServiceImpl(underlying: Logger[Future]) {
  val log: com.twitter.finagle.Service[(String, Int), String] = ...
  val getLogSize: com.twitter.finagle.Service[Unit, Int] = ...
}

Usage in finagle:

val client = ThriftMux.newServiceIface(Logger, "localhost:8080")

client.log(Logger.Log.Args("message", 1)) onSuccess {...}

This avoids the reflection-based initialization (ThriftRichClient) by collecting the relevant types during generation.

Compatibility constructor to build a FutureIface from a ServiceIface:

val loggerFutureIface = Logger.newFutureIface(loggerServiceIface)
loggerFutureIface.log("msg")

Client configuration is done by e.g.

ThriftMux.client.withClientId(ClientId("asdf")).newServiceIface(Logger, dest)

RB_ID=663690
ba43705
@tonyd3 tonyd3 Fix memcached error parsing
Problem
parseErrorMessage only returns the last string, instead of the entire message.
i.e. FAT [20150721-20:26:43.792] memcache: com.twitter.finagle.memcached.protocol.ServerError: object

Solution
Parse all of the tokens instead of the head.

Result
com.twitter.finagle.memcached.protocol.ServerError: out of memory storing object

Signed-off-by: Vladimir Kostyukov <vkostyukov@twitter.com>

RB_ID=719993
d5fae20
@kevinoliver kevinoliver finagle-stats: Fix race condition between latching and snapshots
Problem

RB 717647 introduced a race condition to `MetricsBucketedHistogram`
between latching and snapshots which could lead to snapshots returning
empty data.

1. Snapshot thread, Ts, comes in with current=a and grabs prev=b.
2. Latching thread, Tl, comes in with current=a, sets next=b,
   then does b.clear and sets current=b.
3. Snapshot thread Ts resumes running, but b is now empty.

Solution

Make rolling over to new windows the responsibility of threads calling
into `snapshot`.

Result

Threads should have a safe and consistent view.

RB_ID=719773
d7b94b5
@vkostyukov vkostyukov finagle|util: Use ClassTag instead of ClassManifest
RB_ID=720455
207d515
@kevinoliver kevinoliver finagle-core: Improve BufChannelBuffer.apply
Problem

The construction of a `ChannelBuffer` via `BufChannelBuffer.apply`
for two cases of `Buf`s can be improved.

Solution

 * Remove an unnecessary `ChannelBuffer.slice` in the case of a
   `Buf.ByteArray.Owned`.

 * Add a case for `Buf.ByteBuffer.Owned` and pass its ByteBuffer
   directly to `ChannelBuffers.wrappedBuffer`.

RB_ID=720329
81fbeca
@dschobel dschobel finagle-swift|finagle-serversets: remove bogus netty dep
RB_ID=720848
5500a97
@mariusae mariusae finagle: make Resolve.evalLabeled public
This is useful for protocol implementations outside of Finagle proper.

RB_ID=721075
33df2bd
@luciferous luciferous finagle: rename FailedFast onOrOff to enabled
Problem

`onOrOff` is a confusing name for a boolean parameter.

Solution

Rename it to `enabled`.

RB_ID=720781
TBR=true
USER_HOOK_ARGS=--i-am-evil -x macaw-swift/.hooks/PRESUBMIT/check-swoop
f60e9d8
@vkostyukov vkostyukov Release CSL Stack
RB_ID=721327
65df151