Fix issue #52475: Make loop detector only consider reachable memory #52626

brunocodutra · 2018-07-22T20:30:16Z

As suggested by @oli-obk alloc_ids should be ignored by traversing all Allocations in interpreter memory at a given moment in time, beginning by ByRef locals in the stack.

Generalize the implementation of Hash for EvalSnapshot to traverse Allocations
Generalize the implementation of PartialEq for EvalSnapshot to traverse Allocations
Commit regression tests

Fixes #52626
Fixes #52849

rust-highfive · 2018-07-22T20:30:19Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @petrochenkov (or someone else) soon.

If any changes to this PR are deemed necessary, please add them as extra commits. This ensures that the reviewer can see what has changed since they last reviewed the code. Due to the way GitHub handles out-of-date commits, this should also make it reasonably obvious what issues have or haven't been addressed. Large or tricky changes may require several passes of review and changes.

Please see the contribution instructions for more information.

oli-obk · 2018-07-22T20:48:24Z

I'll have a more detailed look later. Just wondering right now if we can reuse the StableHash stuff of incremental. Seems like we're trying to solve similar issues

brunocodutra · 2018-07-22T21:17:54Z

Just wondering right now if we can reuse the StableHash stuff of incremental.

I'm not sure I follow, but reuse sounds good.

oli-obk · 2018-07-22T23:08:15Z

Hmm alloc_id_recursion_tracker should be removed... noone uses it

So what I meant was reusing

rust/src/librustc/ich/impls_ty.rs

Line 392 in fa60b72

impl<'a> HashStable<StableHashingContext<'a>> for mir::interpret::AllocId {

ecstatic-morse

I think this would be clearer if we moved this subset of fields into its own struct within EvalContext. There's a commit that did this that I reverted in the original PR because those fields were public and I had to change every use of them.

petrochenkov · 2018-07-23T11:02:47Z

r? @oli-obk

brunocodutra · 2018-07-23T18:56:25Z

@ecstatic-morse that's a great point, in fact my idea is was to crawl the transitive allocations in advance as part of constructing EvalContextRef, so that Hash and Eq are trivial.

EDIT: actually I think @oli-obk might be onto something better with HashStable.

brunocodutra · 2018-07-24T07:35:54Z

My understanding of the internals of MIRI are still flaky, so please bear with me if I'm talking nonsense, but after some testing, I'm growing skeptical that the exact example given in #52475 (which I reproduce below) can actually be detected, not because of AllocIds changing, but because the hidden local variables created by &5 seem to imply Frame::locals is an ever growing list. In fact, if I ignore locals that are notByRef in my implementation it does detect this case as a in infinite loop.

#![feature(const_fn, const_let)]

const fn churn_alloc_id() -> usize {
    let mut x: &i32 = &5;
    loop {
        x = &5;
    }
    0
}

fn main() {
    let _ = [(); churn_alloc_id()];
}

Does that make sense?
If so, can anyone come up with an example where AllocIds keep changing, but the number of locals remain constant?

oli-obk · 2018-07-24T07:53:01Z

the hidden local variables created by &5 seem to imply Frame::locals is an ever growing list.

Nope, rustc allocates a single variable and just reuses it.

if you look at the MIR of that function, you can see that it's even more extreme, because rustc generates a hidden static for the &5:

    let mut _0: usize;                   // return place
    scope 1 {
        let mut _1: &i32;                // "x" in scope 1 at src/main.rs:4:9: 4:14
    }
    scope 2 {
    }
    let mut _2: &i32;
    let mut _3: &i32;

    bb0: {                              
        StorageLive(_1);                 // bb0[0]: scope 0 at src/main.rs:4:9: 4:14
        StorageLive(_2);                 // bb0[1]: scope 0 at src/main.rs:4:23: 4:25
        _2 = promoted[1];                // bb0[2]: scope 0 at src/main.rs:4:23: 4:25
                                         // mir::Constant
                                         // + span: src/main.rs:4:23: 4:25
                                         // + ty: &i32
                                         // + literal: promoted[1]
        _1 = _2;                         // bb0[3]: scope 0 at src/main.rs:4:23: 4:25
        StorageDead(_2);                 // bb0[4]: scope 0 at src/main.rs:4:25: 4:26
        goto -> bb1;                     // bb0[5]: scope 1 at src/main.rs:5:5: 7:6
    }

    bb1: {                              
        StorageLive(_3);                 // bb1[0]: scope 1 at src/main.rs:6:13: 6:15
        _3 = promoted[0];                // bb1[1]: scope 1 at src/main.rs:6:13: 6:15
                                         // mir::Constant
                                         // + span: src/main.rs:6:13: 6:15
                                         // + ty: &i32
                                         // + literal: promoted[0]
        _1 = _3;                         // bb1[2]: scope 1 at src/main.rs:6:9: 6:15
        StorageDead(_3);                 // bb1[3]: scope 1 at src/main.rs:6:15: 6:16
        goto -> bb1;                     // bb1[4]: scope 1 at src/main.rs:5:5: 7:6
    }

if you change the code to

    let x = 5;
    let mut x: *const u32 = &x;
    loop {
        let y = 5;
        x = &y;
    }
    0

you get

    let mut _0: usize;                   // return place
    scope 1 {
        let _1: u32;                     // "x" in scope 1 at src/main.rs:4:9: 4:10
        scope 3 {
            let mut _2: *const u32;      // "x" in scope 3 at src/main.rs:5:9: 5:14
            scope 5 {
                let _6: u32;             // "y" in scope 5 at src/main.rs:7:13: 7:14
            }
            scope 6 {
            }
        }
        scope 4 {
        }
    }
    scope 2 {
    }
    let mut _3: *const u32;
    let mut _4: &u32;
    let mut _5: &u32;
    let mut _7: *const u32;
    let mut _8: &u32;
    let mut _9: &u32;

    bb0: {                              
        StorageLive(_1);                 // bb0[0]: scope 0 at src/main.rs:4:9: 4:10
        _1 = const 5u32;                 // bb0[1]: scope 0 at src/main.rs:4:13: 4:14
                                         // ty::Const
                                         // + ty: u32
                                         // + val: Value(ByVal(Bytes(5)))
                                         // mir::Constant
                                         // + span: src/main.rs:4:13: 4:14
                                         // + ty: u32
                                         // + literal: const 5u32
        StorageLive(_2);                 // bb0[2]: scope 1 at src/main.rs:5:9: 5:14
        StorageLive(_3);                 // bb0[3]: scope 1 at src/main.rs:5:29: 5:31
        StorageLive(_4);                 // bb0[4]: scope 1 at src/main.rs:5:29: 5:31
        StorageLive(_5);                 // bb0[5]: scope 1 at src/main.rs:5:29: 5:31
        _5 = &_1;                        // bb0[6]: scope 1 at src/main.rs:5:29: 5:31
        _4 = _5;                         // bb0[7]: scope 1 at src/main.rs:5:29: 5:31
        _3 = move _4 as *const u32 (Misc); // bb0[8]: scope 1 at src/main.rs:5:29: 5:31
        _2 = move _3;                    // bb0[9]: scope 1 at src/main.rs:5:29: 5:31
        StorageDead(_3);                 // bb0[10]: scope 1 at src/main.rs:5:30: 5:31
        StorageDead(_4);                 // bb0[11]: scope 1 at src/main.rs:5:30: 5:31
        StorageDead(_5);                 // bb0[12]: scope 1 at src/main.rs:5:31: 5:32
        goto -> bb1;                     // bb0[13]: scope 3 at src/main.rs:6:5: 9:6
    }

    bb1: {                              
        StorageLive(_6);                 // bb1[0]: scope 3 at src/main.rs:7:13: 7:14
        _6 = const 5u32;                 // bb1[1]: scope 3 at src/main.rs:7:17: 7:18
                                         // ty::Const
                                         // + ty: u32
                                         // + val: Value(ByVal(Bytes(5)))
                                         // mir::Constant
                                         // + span: src/main.rs:7:17: 7:18
                                         // + ty: u32
                                         // + literal: const 5u32
        StorageLive(_7);                 // bb1[2]: scope 5 at src/main.rs:8:13: 8:15
        StorageLive(_8);                 // bb1[3]: scope 5 at src/main.rs:8:13: 8:15
        StorageLive(_9);                 // bb1[4]: scope 5 at src/main.rs:8:13: 8:15
        _9 = &_6;                        // bb1[5]: scope 5 at src/main.rs:8:13: 8:15
        _8 = _9;                         // bb1[6]: scope 5 at src/main.rs:8:13: 8:15
        _7 = move _8 as *const u32 (Misc); // bb1[7]: scope 5 at src/main.rs:8:13: 8:15
        StorageDead(_8);                 // bb1[8]: scope 5 at src/main.rs:8:14: 8:15
        _2 = move _7;                    // bb1[9]: scope 5 at src/main.rs:8:9: 8:15
        StorageDead(_7);                 // bb1[10]: scope 5 at src/main.rs:8:14: 8:15
        StorageDead(_9);                 // bb1[11]: scope 5 at src/main.rs:8:15: 8:16
        StorageDead(_6);                 // bb1[12]: scope 3 at src/main.rs:9:5: 9:6
        goto -> bb1;                     // bb1[13]: scope 3 at src/main.rs:6:5: 9:6
    }

Where you can see that it keeps taking the address of _6 in bb1. There aren't infinite locals, just infinite allocations, because each StorageDead(_6) deletes the allocation, and each StorageLive(_6) creates a new one.

brunocodutra · 2018-07-24T08:07:51Z

Awesome, thanks for the detailed explanation, I'll investigate later today why then non`ByRef` locals seem to be preventing the detection, do you have any clues?

oli-obk · 2018-07-24T08:17:34Z

Most likely because ByVal can mean Value::Scalar(Scalar::Ptr(ptr)) which in turn can again include an AllocId

oli-obk · 2018-07-24T08:19:11Z

src/librustc_mir/interpret/eval_context.rs

+            for local in locals.iter() {
+                // Skip dead locals
+                if let Some(value) = local {
+                    match value {


I think you can just call HashStable::hash_stable on the value to obtain the same effect + simultaneously do the correct thing on ByVal. Have a look at the HashStable impl, it should look very much like your code.

That's what I intend to do, I just wanted a proof of concept first to be sure I understand what's going on

So I tried reusing HashStable for value, however rustc panics:

thread 'main' panicked at 'no value for AllocId', libcore/option.rs:989:5

I guess that means dangling references aren't expected when computing the stable hash, but on the other hand I don't see why that would make the hash not stable. Indeed, hashing the potentially empty result directly does seem to fix the issue.

Does that makes sense?

Yea... That's probably wrong. We should not ICE on

#![feature(const_let)] const FOO: *const u32 = { let x = 42; &x }; fn main() { let z = FOO; }

and similar code. (If we fixed the ICE on the above code, we'd just end up with the ICE you're seeing) So we can just treat dangling pointers as not existing by not hashing them.

Right, so if got what you're saying right, it's safe to fix the implementation of HashStable for AllocId such that it doesn't panic and then just reuse it to hash the Frame::locals here.

brunocodutra · 2018-07-24T08:26:55Z

Sorry, fat fingered the wrong button on my phone.

brunocodutra · 2018-07-24T08:27:25Z

Most likely because ByVal can mean Value::Scalar(Scalar::Ptr(ptr)) which in turn can again include an AllocId

~~Makes total sense, thanks~~

Edit: wait, I think I misunderstood this, isn't what you're saying exactly what line 172 covers?

brunocodutra · 2018-07-24T19:00:29Z

Oh I see now, ByRef vs ByVal, right, totally missed that.

oli-obk

I think this would be clearer if we moved this subset of fields into its own struct within EvalContext.

I think we should do definitely do this. The amount of handwritten equality code is scary. It looks like we'll accidentally break it in the future.

oli-obk · 2018-07-30T08:29:54Z

src/librustc_mir/interpret/eval_context.rs

-    /// steps *until* the loop detector is enabled. When it is positive, it is
-    /// the number of steps after the detector has been enabled modulo the loop
-    /// detector period.
-    pub(crate) steps_since_detector_enabled: isize,


This would murder performance in the regular case. Please undo this commit.

Right, this commit was part of an attempt to detect infinite recursion prior to reaching the stack size limit. The idea was to only snapshot the current frame, which would be much lighter weight, however I soon realized this would obviously lead to false positives and I abandoned the idea, but forgot to revert this commit before pushing this batch.

In any case I have already reverted this on my local workspace.

oli-obk · 2018-07-30T08:31:20Z

src/librustc_mir/interpret/eval_context.rs

    }
 }

-impl<'a, 'b, 'mir, 'tcx, M> HashStable<StableHashingContext<'b>> for EvalSnapshot<'a, 'mir, 'tcx, M>
+impl<'a, 'mir, 'tcx, M> HashStable<StableHashingContext<'a>> for EvalSnapshot<'mir, 'tcx, M>


You should be able to use the hash stable derive macro for this

The macro seems to hardcode the lifetime parameters, is there a straightforward way to extend it for multiple generic parameters?

brunocodutra · 2018-07-30T11:25:30Z

The amount of handwritten equality code is scary

Indeed, I still didn't remove the [WIP] notice because I'm still working out a way to simplify this algorithm, which is proving harder than I first envisioned.

My current idea is to resolve all Allocations reachable through locals in the stack into some custom structure that would be trivial to compara, however I'm kinda caught up on the distinction between the allocations map in TyCtxt and Memory. The implementation of HashStable uses the former, whereas the original infonite loop detection algorithm uses the latter, which one should I use?

brunocodutra · 2018-07-30T11:27:57Z

Also the current implementation pushed is broken, because it doesn't actually clone the Allocations.

oli-obk · 2018-07-30T12:59:08Z

The implementation of HashStable uses the former, whereas the original infonite loop detection algorithm uses the latter, which one should I use?

Oh no. I forgot about this.

Most of the impls are on StableHashingContext instead of on a generic argument. I wonder how many of these impls actually need the context and could be generic instead.

If we can make the impls generic, then we can just use HashStable<Memory>

brunocodutra · 2018-07-30T15:06:58Z

If we can make the impls generic, then we can just use HashStable<Memory>

I'm not sure this would be so straightforward, the allocation map seems to come from a global variable rather than the hashing context, that is, StableHashingContext doesn't seem to require a way to resolve Allocations to be provided.

Based on that observation, my current implementation also resolves allocation using the global context, as opposed to Memory::alloc_map, but I wonder whether this is correct. What's the difference between this two mappings?

oli-obk · 2018-07-30T15:15:37Z

The global context allocations are statics and constants, while the ones in the EvalContext are for computing the current static/constant.

cc @michaelwoerister do you think we can tune HashStable impls by making all the HashStable<StableHashingContext> impls be HashStable<T> with T: HashingContext which exposes a bunch of useful functions

trait HashingContext {
    fn with_allocation<T>(&self, f: impl FnOnce(&Allocation) -> T) -> T;
}

that allow reusing that code?

brunocodutra · 2018-07-30T15:23:30Z

The global context allocations are statics and constants, while the ones in the EvalContext are for computing the current static/constant.

~~I see, makes sense, it turns out we need to use both then.~~

This got me thinking, since the current infinite loop detection algorithm does not snapshot the static memory, I wonder whether it would produce false positives for loops that only change statics. I'll add a regression test for that too.

EDIT: I think I misread this the first time, what do you mean by current static/constant?

brunocodutra · 2018-07-30T22:22:15Z

I reset the branch to a state I'm reasonably confident to be correct. Now I really need help understanding what exactly needs to be done moving forward with the implementation of Eq.

To summarize my understanding, all AllocIds need to be resolved to Allocations through Memory::get(AllocId) starting from Frame::locals. AllocIds can be found both in Scalar::Ptrs as well as Allocation::relocations. This is a different process from that used to compute HashStable, because the latter currently resolves AllocIds to Allocations through the global TyCtxt::alloc_map.

Open questions:

Is the implementation of Eq above correct?
Even though HashStable relies on a different mapping from AllocId to Allocation, is it sufficient for the requirement that x == y implies hash_stable(x) == hash_stable(y), so that generalizing StableHashingContext to make it possible for Memory to implement it may be tracked by a separate issue?

…text

oli-obk · 2018-09-06T14:30:40Z

@bors r+

bors · 2018-09-06T14:30:41Z

📌 Commit 05cdf8d has been approved by oli-obk

bors · 2018-09-06T17:35:47Z

⌛ Testing commit 05cdf8d with merge c318691...

@oli-obk

Fix issue #52475: Make loop detector only consider reachable memory As [suggested](#51702 (comment)) by @oli-obk `alloc_id`s should be ignored by traversing all `Allocation`s in interpreter memory at a given moment in time, beginning by `ByRef` locals in the stack. - [x] Generalize the implementation of `Hash` for `EvalSnapshot` to traverse `Allocation`s - [x] Generalize the implementation of `PartialEq` for `EvalSnapshot` to traverse `Allocation`s - [x] Commit regression tests Fixes #52626 Fixes #52849

bors · 2018-09-06T20:30:54Z

☀️ Test successful - status-appveyor, status-travis
Approved by: oli-obk
Pushing c318691 to master...

RalfJung · 2018-09-19T23:22:44Z

@brunocodutra @oli-obk I must be missing something... I am staring at this code and trying to understand how it avoids going in endless cycles when there are cycles in memory. However, I cannot see anything here that would stop copying when we reach an allocation that has already been copied. Has that been overlooked, or am I just missing the place where the magic is happening?

brunocodutra · 2018-09-20T06:52:56Z

Does stable hashing avoid cycles? I thought it did not and concluded one couldn't form cycles of allocations, at least not in const fn. Can you come up with a counter exemple that hangs the compiler?

RalfJung · 2018-09-20T06:55:08Z

How would stable hashing avoid cycles...? I don't understand how it would even begin to have the infrastructure and state for that.^^

Can you come up with a counter example that hangs the compiler?

I am sure this will be easy once we got a few more features stabilizied. Still trying to get one with the current restricted set.

Another thing: Why does EvalSnapshot::new copy everything with clone, and then later has a ridiculously expensive implementation of PartialEq::eq do compare? Wouldn't it be much better to do the work of EvalSnapshot::snapshot just once, in EvalSnapshot::new?

brunocodutra · 2018-09-20T07:14:07Z

I refer to stable hashing because not only it's part of loop detection, but also pretty much how the snapshot itself is implemented.

As for the other question, I can't think of a good reason besides me being obtuse. I'll revisit this PR later this evening with a fresh mind and much better understanding of the language - this PR was the very first thing I ever wrote in Rust.

RalfJung · 2018-09-20T08:48:12Z

I refer to stable hashing because not only it's part of loop detection, but also pretty much how the snapshot itself is implemented.

Sure but it has nothing to do with detecting cycles.^^

And actually, it's not just cycles. Currently this duplicates allocations when they are referenced multiple times. So imagine

let x = 0;
let x = (&x, &x);
let x = (&x, &x);
let x = (&x, &x);
let x = (&x, &x);
let x = (&x, &x);
let x = (&x, &x);
let x = (&x, &x);
let x = (&x, &x);

This will duplicate the original assertion (just containing 0) 2^n times. So, there definitely needs to be a deduplication. I will open an issue.

As for the other question, I can't think of a good reason besides me being obtuse. I'll revisit this PR later this evening with a fresh mind and much better understanding of the language - this PR was the very first thing I ever wrote in Rust.

That's okay. :) I was just wondering if there is maybe something I missed. I am touching the snapshot stuff in #54380, and I will see if I can change it to snapshot earlier and not copy all of memory. I won't do any deduplication though.

RalfJung · 2018-09-20T09:31:46Z

Hm actually stable hashing does something with tcx. So maybe it has sufficient state to deduplicate something? I don't see where and how this interacts with the snapshotting TBH. At the very least, an extensive comment should be added, answering questions like "why is AllocIdSnapshot an Option?"

RalfJung · 2018-09-20T09:42:44Z

I give up. I tried making a snapshot early, but I cannot comprehend what this code does, and I really should get actual work done.^^

The problem I ran into is that the snapshot isn't actually a snapshot. It's not a deep copy, it's full of references to... somewhere. This definitely needs lots of documentation, right now, the code is incomprehensible as far as I am concerned.

RalfJung · 2018-09-20T10:00:15Z

I opened #54384 to track the lack of documentation. It would be great if you could take care of that @brunoabinader while the memory of this PR is still fresh in your mind :)

And sorry if I came across too complaining. I wanted to get some other work done and the loop detector made that so much more complicated, so I wanted to quickly fix it to be simpler and got frustrated when that didn't work and I couldn't figure out why. I shouldn't have loaded that frustration on you though. I am happy for your contributions!

brunocodutra · 2018-09-20T10:06:26Z

At a high level, what it does is to crawl up the allocation stack and replace every weak reference (AllocId) by a strong shared reference, so that a trivial equality comparison does the right thing, comparing the actual allocations rather than their IDs. I see now why we clone everything and use a reference at the time we compare for equality: it's to avoid duplicating clones. Imagine the same allocation is referred through its AllocId in several places. If we crawl up the allocation structure and replace every AllocId by a clone of the allocation, we will clone the same thing over and over again. Instead we clone all allocations only once and then just map AllocIds to references by the time we need to compare the state for equality. Does that make sense? This is actually not super expensive, the equality comparison needs to crawl up the structure either way, the only extra cost is querying Memory, which is a trade off to avoid duplicating clones.

rust-highfive assigned petrochenkov Jul 22, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 22, 2018

brunocodutra mentioned this pull request Jul 22, 2018

CTFE fails to detect a certain class of infinite loops #52475

Closed

ecstatic-morse reviewed Jul 22, 2018

View reviewed changes

rust-highfive assigned oli-obk and unassigned petrochenkov Jul 23, 2018

oli-obk reviewed Jul 24, 2018

View reviewed changes

brunocodutra closed this Jul 24, 2018

brunocodutra reopened this Jul 24, 2018

oli-obk requested changes Jul 30, 2018

View reviewed changes

brunocodutra added 8 commits September 3, 2018 20:02

Impl Eq and PartialEq for EvalSnapshot in terms of the Snapshot trait

927c709

Add regression test for #52475

806ecab

Add an info log when snapshotting the constant evaluation context

0a84ff0

Add a convenience macro to reduce code duplication

4eb8d94

Keep lines shorter than 100 characters

add9ee2

Use EvalContext's TyCtx for the purpose of hashing the evaluation con…

a26ceb3

…text

Move InfiniteLoopDetector to snapshot.rs

61a999a

Document snapshot.rs

05cdf8d

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Sep 6, 2018

bors merged commit 05cdf8d into rust-lang:master Sep 6, 2018

This was referenced Sep 6, 2018

Report const eval error inside the query #53821

Merged

use NonZeroU32 in newtype_index!macro, change syntax #53315

Merged

brunocodutra deleted the issue-52475 branch September 6, 2018 20:41

RalfJung mentioned this pull request Sep 9, 2018

Remove unchecked_shr/shl from intrinsics rust-lang/miri#451

Merged

RalfJung mentioned this pull request Sep 20, 2018

remove CTFE loop detector #54384

Closed

Fix issue #52475: Make loop detector only consider reachable memory #52626

Fix issue #52475: Make loop detector only consider reachable memory #52626

Conversation

brunocodutra commented Jul 22, 2018 • edited by RalfJung Loading

rust-highfive commented Jul 22, 2018

oli-obk commented Jul 22, 2018

brunocodutra commented Jul 22, 2018

oli-obk commented Jul 22, 2018

ecstatic-morse left a comment

Choose a reason for hiding this comment

petrochenkov commented Jul 23, 2018

brunocodutra commented Jul 23, 2018 • edited Loading

brunocodutra commented Jul 24, 2018 • edited Loading

oli-obk commented Jul 24, 2018

brunocodutra commented Jul 24, 2018 via email

oli-obk commented Jul 24, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brunocodutra commented Jul 24, 2018

brunocodutra commented Jul 24, 2018 • edited Loading

brunocodutra commented Jul 24, 2018

oli-obk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brunocodutra commented Jul 30, 2018

brunocodutra commented Jul 30, 2018 • edited Loading

oli-obk commented Jul 30, 2018 • edited Loading

brunocodutra commented Jul 30, 2018

oli-obk commented Jul 30, 2018

brunocodutra commented Jul 30, 2018 • edited Loading

brunocodutra commented Jul 30, 2018

oli-obk commented Sep 6, 2018

bors commented Sep 6, 2018

bors commented Sep 6, 2018

bors commented Sep 6, 2018

RalfJung commented Sep 19, 2018

brunocodutra commented Sep 20, 2018

RalfJung commented Sep 20, 2018 • edited Loading

brunocodutra commented Sep 20, 2018

RalfJung commented Sep 20, 2018

RalfJung commented Sep 20, 2018 • edited Loading

RalfJung commented Sep 20, 2018 • edited Loading

RalfJung commented Sep 20, 2018

brunocodutra commented Sep 20, 2018 via email • edited Loading

brunocodutra commented Jul 22, 2018 •

edited by RalfJung

Loading

brunocodutra commented Jul 23, 2018 •

edited

Loading

brunocodutra commented Jul 24, 2018 •

edited

Loading

brunocodutra commented Jul 24, 2018 •

edited

Loading

brunocodutra commented Jul 30, 2018 •

edited

Loading

oli-obk commented Jul 30, 2018 •

edited

Loading

brunocodutra commented Jul 30, 2018 •

edited

Loading

RalfJung commented Sep 20, 2018 •

edited

Loading

RalfJung commented Sep 20, 2018 •

edited

Loading

RalfJung commented Sep 20, 2018 •

edited

Loading

brunocodutra commented Sep 20, 2018 via email •

edited

Loading