Support action continuations in the controller #3202

tardieu · 2018-01-19T00:11:57Z

This PR introduces a new type of actions called compositions. Compositions are a kind of sequences with dynamically computed steps. While the components of a sequence must be specified before invocation, components of a composition can be decided as the composition is running.

Like the sequence implementation, the bulk of the composition implementation lies in the controller. It chains the invocations of the component actions in the composition. It keeps track of the execution state. It produces the final activation record for the composition invocation.

In contrast with sequences and in an attempt to keep the changes contained in the controller code, a composition is not a new kind of action but rather a action with a marker annotation. This means no CLI changes for instance.

A conductor action is an action with a defined “conductor” annotation. The value of the annotation does not matter. For simplicity, sequences cannot be conductors. The annotation is ignored on sequences. But sequences may be components of a composition.

Conductor actions drive the execution of compositions. A conductor action may either return a final result or a triplet { action, params, state }. In the latter case, the specified component action is invoked on the specified params object. Upon completion of this action the conductor action is reinvoked with a payload that combines the output of the action with the state returned by the previous conductor invocation. The composition result is the result of the final conductor invocation in the chain of invocations.

A composition is therefore a sequence of invocations alternating invocations of the (fixed) conductor action with invocations of the dynamically specified components of the composition. In a sense, the conductor action acts as a scheduler that decodes and executes a sequential program. Conductor actions make it possible to support dynamic composition in OpenWhisk without committing to any particular execution model (e.g., finite state machines of one kind or another), or invocation conventions.

Compositions may be nested, i.e, a step in a composition may itself be a composition. The controller recognizes this pattern and handles it efficiently.

Like a sequence, one composition is materialized by one action. Creating a composition requires the definition of one conductor action. Deleting the composition consists in deleting the corresponding action. Like a sequence, a composition refers to the composed actions by name (i.e., by reference).

The PR enables a much better implementation of the composer library for OpenWhisk (https://github.com/ibm-functions/composer) that offers faster performance, lower cost (no double billing), and dramatically improves usability. With this PR, a composition is an action. There is no distinction between invoking a composition and an action. Both invocations returns an activation id that can be used to fetch the result of the execution of the action or composition. Both invocations can be blocking. Both can be web actions.

As for sequences, there is a configurable upper bound on the length of a composition (number of steps). In contrast with sequences, the limit is assessed at run time, and the composition invocation is aborted if the limit is exceeded. An entitlement check verify that each composed action is accessible prior to its execution. A composition invocation like a sequence invocation counts as one invocation viz. throttling. In particular, a composition will not be aborted during execution due to throttling. Therefore a composition designed to have fewer steps than the limit will not abort due to quotas once it has started execution.

This PR comes with unit tests for the controller as well as integration tests.

Example:

$ cat > increment.js
function main({ value }) { return { value: value + 1 } }
^D

$ cat > conductor.js
function main(params) {
    switch (params.$step || 0) {
        case 0: delete params.$step; return { params, action: 'increment', state: { $step: 1 } }
        case 1: delete params.$step; return { params, action: 'increment', state: { $step: 2 } }
        case 2: delete params.$step; return params
    }
}
^D

$ wsk action update increment increment.js
$ wsk action update conductor conductor.js -a conductor true

$ wsk action invoke conductor -br -p value 3
{
    "value": 5
}

This example demonstrates a simple composition with two increment steps. The progress of the execution is tracked via the $step field of the parameter object that is propagated from a conductor invocation to the next but hidden from the composed actions.

csantanapr · 2018-01-19T00:29:19Z

worth an email to the mailing list, this is a sweet feature.

csantanapr · 2018-01-19T00:29:32Z

docs? (i.e. .md)

tardieu · 2018-01-19T00:54:35Z

The code is documented. I'll add a docs/compositions.md. But the real power of compositions will come from composer and I am not going to document this here.

…tion compositions

tardieu · 2018-01-19T01:58:59Z

I rewrote the commit history as requested.

cfjedimaster · 2018-01-19T15:03:03Z

Am I wrong to be worried that this may confuse folks with Composer? Like - I assumed this was related and it seems to have some similarities but if it is truly unique, maybe the name "composition" should be changed?

cfjedimaster · 2018-01-19T15:06:04Z

"The annotation is ignored on sequences"
No, no, no, no. Please no. I keep running into the bug where you can't do a default param for sequences and it is silently ignored.

tardieu · 2018-01-19T15:10:59Z

We can support the annotation on sequences. But I would like to do sequences as a separate PR as this one is quite large already.

tardieu · 2018-01-19T15:15:53Z

Composer is being rewritten to take advantage of this PR internally. The composer API is unchanged. The PR makes it possible to execute action compositions efficiently. Composer will continue to help author action compositions by automatically generating the conductor action code from a high-level specification. For example, code equivalent to the double increment conductor.js action code shown above can be generated by Composer from the much simpler code composer.sequence('increment', 'increment').

cfjedimaster · 2018-01-19T15:20:04Z

... interesting. So a user would have 2 main ways to make compositions. Either using Composer or "by hand". Of course, when I work with Composer I'm working with a local file anyway. It feels like this could be - potentially - a bit confusing for the end user. I'm not saying this is bad in anyway, but I worry about how I'd explain this to a group of devs, know what I mean?

Could you elaborate on when I'd use Composer vs compositions? (And since Compser makes compositions, you can see already it's a bit confusing. ;)

tardieu · 2018-01-19T15:24:07Z

Always use Composer! It is much easier, less error prone, and there is no real drawback. There is a good reason I have not written docs for compositions. :D

cfjedimaster · 2018-01-19T15:25:29Z

So if it fair to say: "This PR changes stuff beneath the covers and Ray you can stop paying attention." Cuz I'm totally fine with ignoring this if so. :)

tardieu · 2018-01-19T15:27:42Z

We have been discussing ways to run composer server-side, i.e., upload the composer source directly as opposed to having to compile the composition locally. I suspect we will end up supporting this mode of operation in some way.

tardieu · 2018-01-19T15:34:21Z

Yes you can think of this PR as improving the Composer backend. The conductor action "language" is not really meant for end users. But the PR will have visible positive impact on Composer. First, the distinction between actions and apps will basically disappear. Second, REDIS will be an option (for compositions with large amounts of state) not a requirement.

cfjedimaster · 2018-01-19T15:36:34Z

Thank you - I appreciate you taking the time to explain this to me! (I wanted to do more than just a thumbs up - I really am thankful.)

tardieu · 2018-01-19T15:58:18Z

The distinction between composer and compositions is not gratuitous... This PR with the addition of action compositions is about adding a fundamental capability to the OpenWhisk controller with as little a bias as possible. It does not favor a specific approach to building composition. Composer on the other hand embodies many design choices. For instance, it currently targets Javascript. In contrast, the PR is language/runtime agnostic. While I don't think writing compositions by hand is a compelling proposition for OpenWhisk users, there could be many alternatives to Composer. All would benefit from the PR in the same way.

cfjedimaster · 2018-01-19T16:03:44Z

Ah the JS bias in Composer is a very good point. This would definitely help the non-JS folks.

csantanapr · 2018-01-19T16:21:00Z

@tardieu

The code is documented. I'll add a docs/compositions.md. But the real power of compositions will come from composer and I am not going to document this here.

This is a change that end users would not use directly, but is a change in the primitives to a core/kernel component (controller) to allow more efficient (store and forward state) and language agnostic applications. This is a feature to be use by someone building a framework or tooling for end user (i.e. downstream integrators/providers), but not end users (serverless developers).

So the docs should be written for a downstream integrator/provider, on how the primitive works and some examples how to construct an orchestration to illustrate the concepts. Similar on what you did in your opening comment above.

markusthoemmes

I'd like to clarify on the need of mutability/non-deterministic concurrency methods in the code. This is not a concern of making it pretty but very much a concern of me having had a very hard time following the code around.

markusthoemmes · 2018-02-06T15:15:38Z

core/controller/src/main/scala/whisk/core/controller/actions/PrimitiveActions.scala

+   * @param components the current count of component actions already invoked
+   * @param conductors the current count of conductor actions already invoked
+   */
+  private case class CompositionAccounting(var components: Int = 0, var conductors: Int = 0)


Why is the mutability needed? Could we keep the object immutable and create copies instead? In general that's much more readable and much easier to follow, flow-wise.

Thanks to mutability, it is clear that there is only one CompositionAccounting object ever allocated in the course of the execution of a composition. Therefore there is only one set of counters. With object copies on the other hand, it is hard to keep track of the copies. Why are copies being made, where, when? Are the multiple sets of counters concurrently valid and useful? Moreover, immutability does not prevent forking the object, hence concurrency issues.

Give me ownership types and strong updates... and then I buy your argument. But without ownership control, I am afraid we are discussing style more than safety.

I see what you mean, although I tend to disagree. Having immutable state forces locality of changes vs. being able to pass an object around and magically updating it somewhere else. If you'd want to update state in a method you'd need to make it clear through the return type, that you're returning a new version of that state vs. magically manipulating it elsewhere.

I think though, once the Promises are out of the picture, that notion might become more prominent. I'd be okay focusing on getting the Promise out as the concern is much more around that than around the mutable state.

markusthoemmes · 2018-02-06T15:15:55Z

core/controller/src/main/scala/whisk/core/controller/actions/PrimitiveActions.scala

+                             cause: Option[ActivationId],
+                             var duration: Long,
+                             var maxMemory: Int,
+                             var state: Option[JsObject],


Same comment on mutability.

Same answer. I want one Session object for one session. I believe this one to one mapping makes the code easier to follow. Part of what constitutes a session-its state-changes over the course of the execution, so the corresponding fields of the object are mutable.

If I made this class immutable and used object copies instead as suggested, then the distinction between immutable session elements and mutable session elements would disappear from the type and become buried in the code and comments. I would argue this obfuscates intend and hinder readability.

markusthoemmes · 2018-02-06T15:20:04Z

core/controller/src/main/scala/whisk/core/controller/actions/PrimitiveActions.scala

+                             accounting: CompositionAccounting,
+                             logs: Buffer[ActivationId],
+                             caller: Option[Session],
+                             result: Option[Promise[Either[ActivationId, WhiskActivation]]])


I'll put the comment here, but the concern is around the other functions as well: Do you need to pass a Promise around vs. chaining all actions together using Futures?

When glancing over the code it was very hard to tell for me, where this promise is used and which of the other methods does which side-effects. They all return Unit. There is no way for me to safely tell what is going on if I do not read all of the source-code.

I believe this is a perfect use-case to make use of Futures and chain the methods together and I think the code will very much profit from that. It'll make it virtually impossible to miss cases and/or have dead-ends in the code.

Ok I'll replace the promise with a future.

rabbah · 2018-02-07T03:31:23Z

core/controller/src/main/scala/whisk/core/controller/actions/PrimitiveActions.scala

+                case Failure(t) =>
+                  // parsing failure
+                  Future.successful(ActivationResponse.applicationError(compositionComponentInvalid(next)))
+                case Success(next) =>


could save a level of nesting by doing this:

case Success(next) if (session.accounting.components >= actionSequenceLimit) => Future.successful(ActivationResponse.applicationError(compositionIsTooLong)) case Success(next) => the rest

rabbah · 2018-02-07T03:33:23Z

core/controller/src/main/scala/whisk/core/controller/actions/PrimitiveActions.scala

+    } else {
+      waitForResponse map { timeout =>
+        // blocking invoke, wait until timeout
+        response.withAlternativeAfterTimeout(waitForResponse.get, Future.successful(Left(session.activationId)))


isn't waitForResponse.get == timeout?

yes! will fix

rabbah · 2018-02-07T03:40:52Z

core/controller/src/main/scala/whisk/core/controller/actions/PrimitiveActions.scala

+
+    logging.debug(this, s"recording activation '${activation.activationId}'")
+    WhiskActivation.put(activationStore, activation)(transid, notifier = None) onComplete {
+      case Success(id) => logging.info(this, s"recorded activation")


logging.debug; to follow in general log reduction.

or just drop it entirely, it's logged in the dbstore anyway.

I am confused. The code already invokes logging.debug. Should this change?

line 549:
case Success(id) => logging.info(this, s"recorded activation")

OK I see it.

tardieu · 2018-02-07T05:04:09Z

I agree that the code with a future is simpler than the code with a promise so I am ok with the change. But, FWIW, there was a decent reason to use the more complex concept that I try to summarize below (before I forget).

The future-based implementation is now similar to the sequence implementation. It creates one pending future (ie pending continuation) for each level of nesting (sequence inside a sequence, composition inside a composition). In contrast the promise-based implementation only requires one pending continuation irrespective of nesting. It used instead a call stack (now removed caller field in session object) and unwinds the stack upon termination of a nested composition. Concretely, the callee resumes the execution of the caller explicitly with the promise implementation, whereas with the future implementation all the stack levels simultaneously exist as pending tasks in the scheduler. In short, I believe the future implementation is by nature more expensive. But since we cap the recursion depth to a low value, it is most likely irrelevant in practice.

markusthoemmes · 2018-02-13T17:26:59Z

core/controller/src/main/scala/whisk/core/controller/actions/PrimitiveActions.scala

+          activation.annotations.get("limits").foreach { limitsAnnotation =>
+            limitsAnnotation.asJsObject.getFields("memory") match {
+              case Seq(JsNumber(memory)) =>
+                session.maxMemory = Math.max(session.maxMemory, memory.toInt)


I believe this does not ignore the "not a sequence of number" cases but throws a MatchException.

You're looking for:

activation.annotations.get("limits") .flatMap(_.asJsObject.fields.get("memory")) .flatMap(memory => Try(memory.convertTo[Int]).toOption) .foreach(memory => session.maxMemory = memory max session.maxMemory

To be relatively sure it doesn't throw (asJsObject can still throw)

The memory annotation in activation records is generated. If the annotation is present without the proper type, we should get an internal error. MatchException does that. I don't think silently ignoring the error makes more sense. We might want to have a more specific error message but I don't think this is needed either. The current code follows directly from the SequenceActions implementation.

markusthoemmes · 2018-02-13T17:28:16Z

core/controller/src/main/scala/whisk/core/controller/actions/PrimitiveActions.scala

+        case Right(activation) => // successful invocation
+          session.logs += activation.activationId
+          // activation.duration should be defined but this is not reflected by the type so be defensive
+          session.duration += activation.duration.getOrElse(activation.end.toEpochMilli - activation.start.toEpochMilli)


Maybe go for end - start straight away to avoid confusion?

I am not sure I understand the suggestion here: duration is not equal to end - start and we have to use the correct value. I'll update the comment to be more clear.

markusthoemmes · 2018-02-13T17:28:58Z

core/controller/src/main/scala/whisk/core/controller/actions/PrimitiveActions.scala

+                             action: ExecutableWhiskActionMetaData,
+                             cause: Option[ActivationId],
+                             var duration: Long,
+                             var maxMemory: Int,


Should this be a ByteSize?

OK. Updating.

markusthoemmes · 2018-02-13T17:29:10Z

core/controller/src/main/scala/whisk/core/controller/actions/PrimitiveActions.scala

+                             start: Instant,
+                             action: ExecutableWhiskActionMetaData,
+                             cause: Option[ActivationId],
+                             var duration: Long,


Should this be a FiniteDuration?

No I don't think so. This current type follows from:

case class WhiskActivation(namespace: EntityPath, ... duration: Option[Long] = None)

markusthoemmes

LGTM modulo the comments I included. I'll leave for @rabbah to make the final call

rabbah · 2018-02-15T15:33:16Z

docs/conductors.md

+
+A _conductor action_ is an action with a _conductor_ annotation with a value that is not _falsy_, i.e., a value that is different from zero, null, false, and the empty string.
+
+At this time, sequence actions cannot be conductor actions. The conductor annotation on sequence actions is ignored.


should we phrase this differently? sequence actions are implicit conductors, and adding a conductor annotation has no effect/is ignored?

rabbah · 2018-02-15T15:34:12Z

docs/conductors.md

+
+## Invocation
+
+A conductor action is invoked like a regular action, for instance:


perhaps link to actions.md section on working with actions?

rabbah · 2018-02-15T15:36:24Z

docs/conductors.md

+
+- one activation of the _triple_ action with input `{ value: 3 }` and output `{ value: 9 }`,
+- one activation of the _increment_ action with input `{ value: 9 }` and output `{ value: 10 }`,
+- three _secondary_ activations of the _tripleAndIncrement_ action.


this is not immediately clear:

+3624ad829d4044afa4ad829d40e4af60 tripleAndIncrement +4f91f9ed0d874aaa91f9ed0d87baaa07 tripleAndIncrement

4f9 is the outermost activation record, whereas 362 is the activation of the conductor action itself. This is a little subtle and perhaps warrants a bit more explanation.

i see it below - perhaps foreshadow this explanation then.

tweaking...

rabbah · 2018-02-15T15:38:34Z

docs/conductors.md

+
+Only an internal error (invocation failure or timeout) may result in an even number of derived activations.
+
+### Primary activation


rabbah · 2018-02-15T15:39:17Z

docs/conductors.md

+
+While secondary activations of the conductor action reflect executions of the action code, the primary activation record for the invocation of a conductor action does not. It is a synthetic record similar to the activation record of a sequence action. Concretely in our example, the code in the main function of the _tripleAndIncrement_ conductor action runs exactly three times since there are three secondary activations.
+
+The primary activation record summarizes the series of derived activations:


suggestion: flip primary to come first, secondary section to come second.

rabbah · 2018-02-15T15:54:19Z

docs/conductors.md

+
+There are limits on the number of component action activations and secondary conductor activations in a conductor action invocation. These limits are assessed globally, i.e., if some components of a conductor action invocation are themselves conductor actions, the limits apply to the combined counts of activations across all the conductor action invocations.
+
+The maximum number _n_ of permitted component activations is equal to the maximum number of components in a sequence action. It is configured via the same configuration parameter. The maximum number of secondary conductor activations is _2n+1_.


we should add this to references.md (i dont think we document sequence limit from our past discussion).
we can do that separately.

This commit introduces a new type of actions called conductors. Conductors are a kind of sequence with dynamically computed steps. While the components of a sequence must be specified before invocation, components of a conductor can be decided as the composition is running. Like the sequence implementation, the bulk of the implementation for supporting conductors lies in the controller. It chains the invocations of the component actions. It keeps track of the execution state. It produces the final activation record for the top level conductor invocation. In contrast with sequences and in an attempt to keep the changes contained in the controller code, a conductor is not a new kind of action but rather a action with a marker annotation. This means no CLI changes for instance. A conductor action is an action with a defined "conductor" annotation. The value of the annotation must be a truthy to have effect. The annotation is ignored on sequences. Conductor actions drive the execution of compositions (functions invoked one after the other). A conductor action may either return a final result or a triplet { action, params, state }. In the latter case, the specified component action is invoked on the specified params object. Upon completion of this action the conductor action is reinvoked with a payload that combines the output of the action with the state returned by the previous conductor invocation. The composition result is the result of the final conductor invocation in the chain of invocations. A composition is therefore a sequence of invocations alternating invocations of the (fixed) conductor action with invocations of the dynamically specified components of the composition. In a sense, the conductor action acts as a scheduler that decodes and executes a sequential program. Conductor actions make it possible to support dynamic composition in OpenWhisk without committing to any particular execution model (e.g., finite state machines of one kind or another), or invocation conventions. Compositions may be nested, i.e, a step in a composition may itself be a composition. The controller recognizes this pattern and handles it efficiently. Like a sequence, one composition is materialized by one action. Creating a composition requires the definition of one conductor action. Deleting the composition consists in deleting the corresponding action. Like a sequence, a composition refers to the composed actions by name (i.e., by reference). There is no distinction between invoking a conductor and an action. Both invocations returns an activation id that can be used to fetch the result of the execution of the action or composition. Both invocations can be blocking. Both can be web actions. As for sequences, there is a configurable upper bound on the length of a composition (number of steps). In contrast with sequences, the limit is assessed at run time, and the composition invocation is aborted if the limit is exceeded. An entitlement check verifies that each composed action is accessible prior to its execution. A composition invocation like a sequence invocation counts as one invocation viz. throttling. In particular, a composition will not be aborted during execution due to throttling. Therefore a composition designed to have fewer steps than the limit will not abort due to quotas once it has started execution. This commit comes with unit tests for the controller as well as integration tests. Example: $ cat increment.js function main({ value }) { return { value: value + 1 } } $ cat conductor.js function main(params) { switch (params.$step || 0) { case 0: delete params.$step; return { params, action: 'increment', state: { $step: 1 } } case 1: delete params.$step; return { params, action: 'increment', state: { $step: 2 } } case 2: delete params.$step; return params } } $ wsk action update increment increment.js $ wsk action update conductor conductor.js -a conductor true $ wsk action invoke conductor -r -p value 3 { "value": 5 } This example demonstrates a simple conductor with two increment steps. The progress of the execution is tracked via the $step field of the parameter object that is propagated from a conductor invocation to the next but hidden from the composed actions.

rabbah added controller review Review for this PR has been requested and yet needs to be done. labels Jan 19, 2018

rabbah self-requested a review January 19, 2018 00:26

rabbah self-assigned this Jan 19, 2018

rabbah requested review from markusthoemmes and dubee January 19, 2018 00:27

Add new error messages, annotations, and field names in support of ac…

78ba598

…tion compositions

tardieu force-pushed the conductor branch from 4151163 to 788accd Compare January 19, 2018 01:55

tardieu added 3 commits January 18, 2018 21:46

Add controller logic for action compositions

38c8feb

Add unit tests for action compositions

4964075

Add integration tests for action compositions

639c495

tardieu force-pushed the conductor branch from 788accd to 639c495 Compare January 19, 2018 02:48

rabbah changed the title ~~Action compositions~~ Support action continuations in the controller Jan 19, 2018

tardieu added 2 commits February 5, 2018 20:19

Address review comments

56fa569

Documentation tweaks

4269eae

markusthoemmes requested changes Feb 6, 2018

View reviewed changes

rabbah removed the request for review from dubee February 6, 2018 17:43

tardieu added 2 commits February 6, 2018 21:05

Replace promise with futures

7014267

Refactoring: make only one call to completeActivation

f89b7ff

rabbah reviewed Feb 7, 2018

View reviewed changes

tardieu added 2 commits February 6, 2018 23:15

More suggested code tweaks

9885236

Drop caller in session since it is no longer needed

073fca4

tardieu added 5 commits February 8, 2018 11:54

Merge branch 'master' into conductor

0d758f9

Rename compositions.md to conductors.md

d0e086f

Delete compositions.md

3cb2a2f

Merge branch 'master' into conductor

f913980

Update conductor action documentation

0b10eef

markusthoemmes reviewed Feb 13, 2018

View reviewed changes

markusthoemmes approved these changes Feb 13, 2018

View reviewed changes

tardieu added 2 commits February 13, 2018 13:38

Change maxMemory session field type to ByteSize

a6d0b64

Merge branch 'master' into conductor

6a129ce

rabbah reviewed Feb 15, 2018

View reviewed changes

rabbah approved these changes Feb 15, 2018

View reviewed changes

Making requested changes to conductor documentation

6bceae6

rabbah merged commit 78105f9 into apache:master Feb 15, 2018

starpit mentioned this pull request Feb 19, 2018

update to composer "v2" ibm-functions/shell#470

Closed


		A _conductor action_ is an action with a _conductor_ annotation with a value that is not _falsy_, i.e., a value that is different from zero, null, false, and the empty string.

		At this time, sequence actions cannot be conductor actions. The conductor annotation on sequence actions is ignored.


		## Invocation

		A conductor action is invoked like a regular action, for instance:


		Only an internal error (invocation failure or timeout) may result in an even number of derived activations.

		### Primary activation


		While secondary activations of the conductor action reflect executions of the action code, the primary activation record for the invocation of a conductor action does not. It is a synthetic record similar to the activation record of a sequence action. Concretely in our example, the code in the main function of the _tripleAndIncrement_ conductor action runs exactly three times since there are three secondary activations.

		The primary activation record summarizes the series of derived activations:


		There are limits on the number of component action activations and secondary conductor activations in a conductor action invocation. These limits are assessed globally, i.e., if some components of a conductor action invocation are themselves conductor actions, the limits apply to the combined counts of activations across all the conductor action invocations.

		The maximum number _n_ of permitted component activations is equal to the maximum number of components in a sequence action. It is configured via the same configuration parameter. The maximum number of secondary conductor activations is _2n+1_.

Support action continuations in the controller #3202

Support action continuations in the controller #3202

Conversation

tardieu commented Jan 19, 2018

csantanapr commented Jan 19, 2018

csantanapr commented Jan 19, 2018

tardieu commented Jan 19, 2018

tardieu commented Jan 19, 2018

cfjedimaster commented Jan 19, 2018

cfjedimaster commented Jan 19, 2018

tardieu commented Jan 19, 2018

tardieu commented Jan 19, 2018

cfjedimaster commented Jan 19, 2018

tardieu commented Jan 19, 2018

cfjedimaster commented Jan 19, 2018

tardieu commented Jan 19, 2018

tardieu commented Jan 19, 2018

cfjedimaster commented Jan 19, 2018

tardieu commented Jan 19, 2018

cfjedimaster commented Jan 19, 2018

csantanapr commented Jan 19, 2018 • edited Loading

markusthoemmes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tardieu Feb 6, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rabbah Feb 7, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tardieu commented Feb 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markusthoemmes left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

csantanapr commented Jan 19, 2018 •

edited

Loading

tardieu Feb 6, 2018 •

edited

Loading

rabbah Feb 7, 2018 •

edited

Loading