Use non-atomic flushing with block replay #10148

sipa · 2017-04-04T09:50:50Z

This implements an alternative solution to the flush-time memory usage peak, suggested by @gmaxwell.

Instead of relying on using atomic batch writes in LevelDB for the chainstate, we rely on the fact that we have an external log of updates to it already (called the blockchain).

This patch adds an extra "head blocks" to the chainstate, which gives the range of blocks for writes may be incomplete. At the start of a flush, we write this record, write the dirty dbcache entries in 16 MiB batches, and at the end we remove the heads record again. If it is present at startup it means we crashed during flush, and we rollback/roll forward blocks inside of it to get a consistent tip on disk before proceeding.

If a flush completes succesfully, the resulting database is compatible with previous versions (down to 0.8). If the node crashes in the middle of a flush, a version of the code with this patch is needed to recovery.

sipa · 2017-04-04T09:55:57Z

This badly needs testing, but I'm not sure how to simulate crashes in the middle of flushing (I've manually verified this patch can recover from failure by introducing an exit(0) in the middle of the flush code).

laanwj · 2017-04-04T10:43:40Z

Cool!

but I'm not sure how to simulate crashes in the middle of flushing

I'll get to that :)

sipa · 2017-04-05T09:38:05Z

Rebased, fixed a bug, and added a commit to allows simulating crashes after partial flushes.

laanwj · 2017-04-05T09:41:08Z

src/txdb.cpp

+                static FastRandomContext rng;
+                if (rng.rand32() % crash_simulate == 0) {
+                    LogPrintf("Simulating a crash. Goodbye.");
+                    sync();


This sync should be optional at least (not a realistic crash otherwise)

I've just removed the sync. It seems not needed for testing (I've done reindexes with hundreds of crashes in between).

gmaxwell · 2017-04-05T16:28:54Z

contrib/devtools/check-doc.py is unhappy that you added new arguments without asking for permission from the argument gods.

gmaxwell · 2017-04-05T16:31:53Z

src/txdb.cpp

 bool CCoinsViewDB::BatchWrite(CCoinsMap &mapCoins, const uint256 &hashBlock) {
    CDBBatch batch(db);
    size_t count = 0;
    size_t changed = 0;
+    size_t batch_size = (size_t)GetArg("-dbbatchsize", nDefaultDbBatchSize) << 20;


If we have memory usage that is some multiple of this, perhaps the argument should be in the form of the actual usage rather than the batch size?

Well the relevant constraint is the memory usage peak from allocating the batch, which depends on the batch memory usage, not dbcache memory usage. Also, I don't think anyone will need to change this property (except for tests, where it's very useful to get much more frequent partial flushes).

TheBlueMatt · 2017-04-11T14:01:18Z

src/dbwrapper.h

@@ -75,6 +83,7 @@ class CDBBatch
        leveldb::Slice slValue(ssValue.data(), ssValue.size());

        batch.Put(slKey, slValue);
+        size_estimate += 3 + slKey.size() + slValue.size();


Can you add comments as to why 3 (and, below, 2) bytes are overhead here?

TheBlueMatt · 2017-04-11T18:10:16Z

src/init.cpp

+                    strLoadError = _("Unable to replay blocks. You will need to rebuild the databse using -reindex.");
+                    break;
+                }
+                pcoinsTip->SetBestBlock(pcoinsdbview->GetBestBlock()); // TODO: only initialize pcoinsTip after ReplayBlocks


Hmm, yes. Were you intending to do this after this PR? Can you just delete it and re-create it here? I feel like it may make sense to move the chainActive.Tip-setting from LoadBlockIndexDB to after this point.

Speaking of which, did you mean to add a PruneBlockIndexCandidates() to ReplayBlocks ala LoadBlockIndexDB?

Were you intending to do this after this PR?

Yes, I tried doing it inside the PR, but doing it properly requires a bit more shuffling around and refactoring, which I'd prefer to keep for later.

Speaking of which, did you mean to add a PruneBlockIndexCandidates() to ReplayBlocks ala LoadBlockIndexDB?

ReplayBlocks doesn't touch the block index, so I don't think that would have any effect.

The PruneBlockIndexCandidates in LoadBlockIndexDB uses chainActive.Tip(), so I assumed it may need to be re-run with the new tip (though likely not a bug without it, just a should-do). I'm ok with cleaning this stuff up in a followup PR, but it seems less than ideal as-is right now.

TheBlueMatt · 2017-04-11T18:37:57Z

src/validation.cpp

+    auto pindexUpto = mapBlockIndex[hashUpto];
+
+    int nHeight = 1; // Skip the genesis block
+    if (mapBlockIndex.count(hashBest)) {


Should we also fail here if hashBest has been written (ie is non-IsNull) but isnt in mapBlockIndex?

I think that would be caught by other code we already have, but I've added it here.

TheBlueMatt · 2017-04-11T18:53:29Z

src/validation.cpp

+                inputs.ModifyCoins(txin.prevout.hash)->Spend(txin.prevout.n);
+            }
+        }
+        inputs.ModifyNewCoins(tx->GetHash(), tx->IsCoinBase())->FromTx(*tx, nHeight);


Based on the comment above ModifyNewCoins I do not believe this works, we may need something new to capture the "maybe not fresh, but definitely fully overwrite in any case" case.

Nice catch, fixed.

TheBlueMatt · 2017-04-11T18:55:23Z

src/validation.cpp

+        ReplayBlock(block, cache, pindex->nHeight);
+    }
+    cache.SetBestBlock(hashUpto);
+    chainActive.SetTip(pindexUpto);


It seems super weird to be acting entirely on non-globals, and then suddenly set a global here.

Good point, fixed.

sipa · 2017-04-12T10:42:56Z

Addressed some of @TheBlueMatt's comments.

TheBlueMatt · 2017-04-12T14:04:48Z

src/coins.h

@@ -315,6 +315,9 @@ class CCoinsView
    //! Retrieve the block hash whose state this CCoinsView currently represents
    virtual uint256 GetBestBlock() const;

+    //! Retrieve the block hash up to which changes are included


s/which changes/which some changes/?

TheBlueMatt · 2017-04-12T14:14:22Z

src/validation.cpp

+    if (mapBlockIndex.count(hashBest)) {
+        auto pindexBest = mapBlockIndex[hashBest];
+        if (pindexUpto->GetAncestor(pindexBest->nHeight) != pindexBest) {
+            return error("ReplayBlocks(): chainstate tip does not derive from final boundary");


I believe we'll hit this if we ever crash during a disconnect? Seems kinda annoying to not support disconnect.

Good catch. ReplayBlocks should learn to deal with a reorg.

TheBlueMatt · 2017-04-12T14:21:03Z

src/txdb.cpp

 bool CCoinsViewDB::BatchWrite(CCoinsMap &mapCoins, const uint256 &hashBlock) {
    CDBBatch batch(db);
    size_t count = 0;
    size_t changed = 0;
+    size_t batch_size = (size_t)GetArg("-dbbatchsize", nDefaultDbBatchSize) << 20;
+    if (!hashBlock.IsNull()) {
+        batch.Write(DB_BEST_BLOCK_UPTO, hashBlock);


It seems to me this API ties us to only doing batches per-block, and never across long chains of actions (or at least not across multiple reorgs). Consider the case where you disconnect A to get to B, then disconnect B to get to C then connect D. There is no way to encode that you need to ensure everything from disconnecting B must be replayed to ensure there are no leftover entries from that, I believe. This is likely OK, but should likely be documented somewhere to ensure we dont end up adding a multi-reorg-flush bug later on.

That's a great point. I hadn't considered that in the case of a reorg the set of partially written changes may include things from multiple branches. It seems solvable by allowing the 'upto' blocks to be a list of tip hashes, and then at start time choose which ones to undo and which ones to replay. I think that's a problem for later, but it makes sense to have a comment about it.

sipa · 2017-04-16T14:18:16Z

Updated to deal with reorganizations. The disk format and recovery code can now also deal with multiple partially written branches. That functionality is not needed yet, but means we can switch to different partial flushing strategies later without breaking compatibility with older versions.

TheBlueMatt · 2017-04-17T16:01:59Z

src/init.cpp

+                    strLoadError = _("Unable to replay blocks. You will need to rebuild the database using -reindex-chainstate.");
+                    break;
+                }
+                pcoinsTip->SetBestBlock(pcoinsdbview->GetBestBlock()); // TODO: only initialize pcoinsTip after ReplayBlocks


I think you can do this for (almost) free now. See TheBlueMatt@747b766, though if you dont want to take it here I'll just PR it afterwards.

TheBlueMatt · 2017-04-17T16:44:53Z

src/validation.cpp

+    if (blockUndo.vtxundo.size() + 1 != block.vtx.size())
+        return error("RollbackBlock(): block and undo data inconsistent");
+
+    for (size_t i = 0; i < block.vtx.size(); ++i) {


Doesn't this need to be in reverse order (like in DisconnectBlock, maybe you should just go ahead and add an option to DisconnectBlock to ignore errors in a pervious commit to make it easier to review?)?

Good catch, fixed.

I'll try to do the merging in a extra commit.

TheBlueMatt · 2017-04-17T16:56:43Z

src/validation.cpp

@@ -1505,18 +1505,15 @@ bool ApplyTxInUndo(const CTxInUndo& undo, CCoinsViewCache& view, const COutPoint
    CCoinsModifier coins = view.ModifyCoins(out.hash);
    if (undo.nHeight != 0) {
        // undo data contains height: this is the last output of the prevout tx being spent
-        if (!coins->IsPruned())
-            fClean = fClean && error("%s: undo data overwriting existing transaction", __func__);
+        if (!coins->IsPruned()) fClean = false;


Would be nice to not lose the error messages by adding a flag for printing errors. I think you need the flag either way for the next line, as I dont think you can run the Clear() if we're re-undoing a tx.

There is no change in behavior here; the error case doesn't cause a return from the function. I believe the new (and existing) code is fine: if a failure is detected, the caller (VerifyDB or DisconnectBlock) won't flush the changes view to the level below, ignoring the resulting inconsistent state.

If you insist, I'll add a flag to ignore the error messages, but (perhaps in a separate PR) I think we should rid of these error() calls and instead perhaps return some return code. Reporting of these things doesn't belong here.

I just find the errors useful to keep around, my real concern is the Clear() one line down, which I believe is an actual bug for the new usage in RollbackBlock.

I'm not convinced the Clear() is wrong - it does mean we're passing over a state where the outputs for that TX were all full spent. However, this is hard to reason about, and you may well be right. Furthermore, it seems that Clear() has no purpose. In the 'clean' case, the output is already pruned, to the Clear() is a no-op. In the other case it doesn't matter. I'm removing it.

Hmm, you may be right. Indeed, however, hard to reason about.

TheBlueMatt · 2017-04-17T18:44:54Z

src/validation.cpp

+    std::set<const CBlockIndex*, CBlockIndexWorkComparator> vpindexRollback;
+    for (size_t i = 1; i < pindexHeads.size(); ++i) {
+        const CBlockIndex *pindexHead = pindexHeads[i];
+        while (fGenesis ? pindexHead != nullptr : pindexHead != pindexFork) {


I'm not convinced this is right. What if you connect both A and B, in the simple case? Now you'll disconnect B before you re-connect A and then re-connect B? Is that neccessary?

Note that i starts at 1 in the loop, skipping the branch that leads to the new tip. I've added a comment to clarify.

Indeed, but that only means you wont do the disconnect-then-reconnect thing for one block (which I suppose may be fine for this PR), but you will do it if you have two back-to-back blocks in the list (or am I confused?).

No. Let's say you have a chain A<-B<-C that was being flushed (meaning the old tip was A, you crashed in the middle of writing the changes for B and C, with C the intended new tip). In this case, at recovery time, GetBlockHeads() will return [C,A]. pindexFork will be A. The loop above will only process A, but because A is already the fork point, nothing is added to the disconnect set.

Yes, OK, it wasnt clear to me what GetBlockHeads() should be returning there, Flush seemed to indicate something different.

sipa

Thanks for the detailed review, @TheBlueMatt.

sipa · 2017-04-17T18:57:46Z

src/validation.cpp

@@ -1505,18 +1505,15 @@ bool ApplyTxInUndo(const CTxInUndo& undo, CCoinsViewCache& view, const COutPoint
    CCoinsModifier coins = view.ModifyCoins(out.hash);
    if (undo.nHeight != 0) {
        // undo data contains height: this is the last output of the prevout tx being spent
-        if (!coins->IsPruned())
-            fClean = fClean && error("%s: undo data overwriting existing transaction", __func__);
+        if (!coins->IsPruned()) fClean = false;


There is no change in behavior here; the error case doesn't cause a return from the function. I believe the new (and existing) code is fine: if a failure is detected, the caller (VerifyDB or DisconnectBlock) won't flush the changes view to the level below, ignoring the resulting inconsistent state.

If you insist, I'll add a flag to ignore the error messages, but (perhaps in a separate PR) I think we should rid of these error() calls and instead perhaps return some return code. Reporting of these things doesn't belong here.

sipa · 2017-04-17T19:07:02Z

src/validation.cpp

+    if (blockUndo.vtxundo.size() + 1 != block.vtx.size())
+        return error("RollbackBlock(): block and undo data inconsistent");
+
+    for (size_t i = 0; i < block.vtx.size(); ++i) {


Good catch, fixed.

I'll try to do the merging in a extra commit.

sipa · 2017-04-17T19:07:43Z

src/validation.cpp

+    std::set<const CBlockIndex*, CBlockIndexWorkComparator> vpindexRollback;
+    for (size_t i = 1; i < pindexHeads.size(); ++i) {
+        const CBlockIndex *pindexHead = pindexHeads[i];
+        while (fGenesis ? pindexHead != nullptr : pindexHead != pindexFork) {


Note that i starts at 1 in the loop, skipping the branch that leads to the new tip. I've added a comment to clarify.

NicolasDorier · 2017-04-18T05:52:01Z

src/txdb.cpp

+        std::vector<uint256> heads = GetHeadBlocks();
+        // Construct a set with all existing heads, excluding the new tip.
+        std::set<uint256> setHeads(heads.begin(), heads.end());
+        if (setHeads.empty() || !tip.IsNull()) setHeads.insert(tip);


can tip be null ? I would have expected it is at least genesis.

Tip can be null when it's the first write ever to the database.

I am surprised, I always thought the coinbase of the genesis block was not added in the database. This would mean that having it to null to mean "before processing the genesis", is the same than enforcing the best block to be at least the genesis block.

NicolasDorier · 2017-04-18T06:05:15Z

src/validation.cpp

+
+    // Find last common ancestor of all heads.
+    for (const uint256& hash : hashHeads) {
+        if (hash.IsNull()) {


I am confused about all this fGenesis variable. I guess this is related to my other comment on 02dfa4a#r111875593

I would not expect, pindexFork to be null, as the work fork point should be the genesis block.

I've added a lengthy comment to clarify.

This requires that we not access pcoinsTip in InitBlockIndex's FlushStateToDisk (so we just skip it until later in AppInitMain) and the LoadChainTip in LoadBlockIndex (which there is already one later in AppinitMain, after ReplayBlocks, so skipping it there is fine). Includes some simplifications by Suhas Daftuar and Pieter Wuille.

sipa · 2017-06-26T18:46:29Z

Rebased, and squashed the last two commits.

sipa · 2017-06-26T20:16:00Z

Travis where art thou

Adds new functional test, dbcrash.py, which uses -dbcrashratio to exercise the logic for recovering from a crash during chainstate flush. dbcrash.py is added to the extended tests, as it may take ~10 minutes to run Use _Exit() instead of exit() for crash simulation This eliminates stderr output such as: terminate called without an active exception or Assertion failed: (!pthread_mutex_destroy(&m)), function ~recursive_mutex, file /usr/local/include/boost/thread/pthread/recursive_mutex.hpp, line 104. Eliminating the stderr output on crash simulation allows testing with test_runner.py, which reports a test as failed if stderr is produced.

sdaftuar · 2017-06-27T21:06:51Z

re-ACK 176c021

laanwj · 2017-06-28T16:25:17Z

Tested ACK 176c021

176c021 [qa] Test non-atomic chainstate writes (Suhas Daftuar) d6af06d Dont create pcoinsTip until after ReplayBlocks. (Matt Corallo) eaca1b7 Random db flush crash simulator (Pieter Wuille) 0580ee0 Adapt memory usage estimation for flushing (Pieter Wuille) 013a56a Non-atomic flushing using the blockchain as replay journal (Pieter Wuille) b3a279c [MOVEONLY] Move LastCommonAncestor to chain (Pieter Wuille) Tree-SHA512: 47ccc62303f9075c44d2a914be75bd6969ff881a857a2ff1227f05ec7def6f4c71c46680c5a28cb150c814999526797dc05cf2701fde1369c06169f46eccddee

TheBlueMatt · 2017-07-07T19:56:21Z

utACK-sans-tests once the fixes for init-order bugs here goes through in #10758.

morcos · 2017-07-20T16:09:56Z

posthumous utACK-sans-tests and modulo some of the same bugs @TheBlueMatt found and fixed in #10758. I'll review that now.

176c021 [qa] Test non-atomic chainstate writes (Suhas Daftuar) d6af06d Dont create pcoinsTip until after ReplayBlocks. (Matt Corallo) eaca1b7 Random db flush crash simulator (Pieter Wuille) 0580ee0 Adapt memory usage estimation for flushing (Pieter Wuille) 013a56a Non-atomic flushing using the blockchain as replay journal (Pieter Wuille) b3a279c [MOVEONLY] Move LastCommonAncestor to chain (Pieter Wuille) Tree-SHA512: 47ccc62303f9075c44d2a914be75bd6969ff881a857a2ff1227f05ec7def6f4c71c46680c5a28cb150c814999526797dc05cf2701fde1369c06169f46eccddee

aab15d7 ReplayBlocks: use find instead of brackets operator to access to the element. (furszy) e898353 [Refactoring] Use const CBlockIndex* where appropriate (random-zebra) c76fa04 qa: Extract rpc_timewait as test param (furszy) 0f832e3 shutdown: Stop threads before resetting ptrs (MarcoFalke) 67aebbf http: Remove numThreads and ThreadCounter (Wladimir J. van der Laan) e24c710 http: Remove WaitExit from WorkQueue (Wladimir J. van der Laan) b8f7364 http: Join worker threads before deleting work queue (Wladimir J. van der Laan) 7d68769 rpc: further constrain the libevent workaround (Cory Fields) 75af065 rpc: work-around an upstream libevent bug (Cory Fields) 50e5833 Always return true if AppInitMain got to the end (Matt Corallo) bd70dcc [qa] Test non-atomic chainstate writes (furszy) 8f04970 Dont create pcoinsTip until after ReplayBlocks. (Matt Corallo) 93f2b15 Random db flush crash simulator (Pieter Wuille) 72f3b17 Adapt memory usage estimation for flushing (Pieter Wuille) 8540113 Non-atomic flushing using the blockchain as replay journal (Pieter Wuille) 8d6625f [MOVEONLY] Move LastCommonAncestor to chain (Pieter Wuille) Pull request description: > This patch adds an extra "head blocks" to the chainstate, which gives the range of blocks for writes may be incomplete. At the start of a flush, we write this record, write the dirty dbcache entries in 16 MiB batches, and at the end we remove the heads record again. If it is present at startup it means we crashed during flush, and we rollback/roll forward blocks inside of it to get a consistent tip on disk before proceeding. > If a flush completes succesfully, the resulting database is compatible with previous versions. If the node crashes in the middle of a flush, a version of the code with this patch is needed to recovery. An adaptation of the following PRs with further modifications to the `feature_dbcrash.py` test to be up-to-date with upstream and solve RPC related bugs. * bitcoin#10148. * Increase RPC wait time. * bitcoin#11831 * bitcoin#11593 * bitcoin#12366 * bitcoin#13837 * bitcoin#13894 ACKs for top commit: random-zebra: ACK aab15d7 Fuzzbawls: ACK aab15d7 Tree-SHA512: 898806746f581a9eb8deb0155c558481abf4454c6f3b3c3ad505c557938d0700fe9796e98e36492286ae869378647072c3ad77ad65e9dd7075108ff96469ade1

sipa force-pushed the non_atomic_flush branch from 4b29c9a to 2c3114a Compare April 4, 2017 10:30

laanwj added UTXO Db and Indexes Resource usage labels Apr 4, 2017

sipa force-pushed the non_atomic_flush branch from 2c3114a to aa249d7 Compare April 5, 2017 09:36

laanwj reviewed Apr 5, 2017

View reviewed changes

sipa force-pushed the non_atomic_flush branch from aa249d7 to 3b41e6c Compare April 5, 2017 12:27

gmaxwell reviewed Apr 5, 2017

View reviewed changes

sipa force-pushed the non_atomic_flush branch from 3b41e6c to a4ddc70 Compare April 6, 2017 07:39

TheBlueMatt reviewed Apr 11, 2017

View reviewed changes

sipa force-pushed the non_atomic_flush branch 2 times, most recently from 22bb19a to 16fc013 Compare April 12, 2017 10:40

sipa mentioned this pull request Apr 12, 2017

Switch chainstate db and cache to per-txout model #10195

Merged

TheBlueMatt reviewed Apr 12, 2017

View reviewed changes

sipa force-pushed the non_atomic_flush branch 2 times, most recently from 87d5f62 to 00c29e7 Compare April 16, 2017 14:16

sipa force-pushed the non_atomic_flush branch from 00c29e7 to c9066bc Compare April 16, 2017 16:45

TheBlueMatt reviewed Apr 17, 2017

View reviewed changes

sipa commented Apr 17, 2017

View reviewed changes

sipa force-pushed the non_atomic_flush branch 2 times, most recently from 8bdc5cd to 1c41a6e Compare April 17, 2017 19:57

sipa changed the title ~~Use non-atomic flushing with block replay~~ [WIP] Use non-atomic flushing with block replay Apr 17, 2017

NicolasDorier reviewed Apr 18, 2017

View reviewed changes

sipa and others added 3 commits June 26, 2017 10:45

Adapt memory usage estimation for flushing

0580ee0

Random db flush crash simulator

eaca1b7

sipa force-pushed the non_atomic_flush branch from 03604ea to 56bd849 Compare June 26, 2017 18:18

sipa force-pushed the non_atomic_flush branch from 56bd849 to 176c021 Compare June 26, 2017 21:49

laanwj merged commit 176c021 into bitcoin:master Jun 28, 2017

sdaftuar mentioned this pull request Jun 28, 2017

Bound memory usage when recovering from a crash during chainstate write #10693

Closed

jnewbery mentioned this pull request Jun 29, 2017

[tests] nits in dbcrash.py #10704

Merged

jnewbery mentioned this pull request Jul 31, 2017

TODO for release notes 0.15.0 #9889

Closed

12 tasks

dagurval mentioned this pull request May 31, 2018

Use non-atomic flushing with block replay bitcoinxt/bitcoinxt#439

Merged

sdaftuar mentioned this pull request Jul 1, 2019

ci: Add Travis check to make sure unit test coverage reports stay deterministic #16320

Closed

PastaPastaPasta mentioned this pull request Jul 17, 2019

Backports 0.15 pr21 dashpay/dash#3029

Merged

furszy mentioned this pull request Feb 16, 2021

Use non-atomic flushing with block replay PIVX-Project/PIVX#2207

Merged

bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022

Use non-atomic flushing with block replay #10148

Use non-atomic flushing with block replay #10148

Conversation

sipa commented Apr 4, 2017 • edited Loading

sipa commented Apr 4, 2017 • edited Loading

laanwj commented Apr 4, 2017

sipa commented Apr 5, 2017

laanwj Apr 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gmaxwell commented Apr 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sipa commented Apr 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sipa commented Apr 16, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sipa Apr 17, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sipa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NicolasDorier Apr 20, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sipa Apr 18, 2017 • edited Loading

Choose a reason for hiding this comment

sipa commented Jun 26, 2017

sipa commented Jun 26, 2017

sdaftuar commented Jun 27, 2017

laanwj commented Jun 28, 2017

TheBlueMatt commented Jul 7, 2017

morcos commented Jul 20, 2017

sipa commented Apr 4, 2017 •

edited

Loading

sipa commented Apr 4, 2017 •

edited

Loading

laanwj Apr 5, 2017 •

edited

Loading

sipa Apr 17, 2017 •

edited

Loading

NicolasDorier Apr 20, 2017 •

edited

Loading

sipa Apr 18, 2017 •

edited

Loading