wallet: Add missing cs_wallet/cs_KeyStore locks to wallet #11634

practicalswift · 2017-11-08T09:33:55Z

Add missing wallet locks:

Calling the function GetConflicts(...) requires holding the mutex cs_wallet
Calling the function IsSpent(...) requires holding the mutex cs_wallet
Accessing the variables mapKeys and mapCryptedKeys requires holding the mutex cs_KeyStore
Accessing the variable nTimeFirstKey requires holding the mutex cs_wallet
Accessing the variable mapWallet requires holding the mutex cs_wallet
Accessing the variable nTimeFirstKey requires holding the mutex cs_wallet

promag · 2017-11-08T13:39:00Z

src/wallet/wallet.cpp

@@ -3953,6 +3960,7 @@ CWallet* CWallet::CreateWalletFromFile(const std::string walletFile)
            for (const CWalletTx& wtxOld : vWtx)
            {
                uint256 hash = wtxOld.GetHash();
+                LOCK(walletInstance->cs_wallet);


Either:

lock once before the loop;

loop first with the lock to mapWallet.find(hash) for all vWtx and then loop again without the lock to walletdb.WriteTx();

lock only the mapWallet.find(hash).

practicalswift · 2017-11-08T13:53:40Z

@promag Thanks for reviewing! Feedback addressed. Looks good? :-)

promag · 2017-11-08T15:08:04Z

src/wallet/wallet.cpp

@@ -3950,6 +3957,7 @@ CWallet* CWallet::CreateWalletFromFile(const std::string walletFile)
        {
            CWalletDB walletdb(*walletInstance->dbw);

+            LOCK(walletInstance->cs_wallet);


This should be before the line above. Anyway, there is no need to lock this wallet since it's a fresh instance, unknown to the remaining system. Unless you are adding the lock because of other asserts and if so please commit them to justify this change?

Now moved before the line above. Without that lock we'll trigger Clang thread safety analysis warnings when #11634 is merged due to:

src/wallet/wallet.h: std::map<uint256, CWalletTx> mapWallet GUARDED_BY(cs_wallet);

As agreed upon in #11226 (comment) all guard/lock annotations are added in #11634 and all extra locking is submitted as separate PR:s (such as this one).

The annotations are compile-time only whereas the locks change run-time behaviour (and thus needs extra scrunity!).

As agreed upon in #11226 (comment) all guard/lock annotations are added in #11634 and all extra locking is submitted as separate PR:s (such as this one)

I think you swapped the PR numbers?

Yes, you're right - the annotations are added in #11226 :-)

promag · 2017-11-08T20:36:25Z

utACK 007fcbf.

* Reading the variables mapTxSpends and mapWallet (via IsSpent(...) call) require holding the mutex cs_wallet. * Reading the variables mapKeys and mapCryptedKeys require holding the mutex cs_KeyStore. * Reading the variable nTimeFirstKey requires holding the mutex cs_wallet. * Reading the variable mapWallet requires holding the mutex cs_wallet. Github-Pull: bitcoin#11634 Rebased-From: 007fcbff2fe5cd4798721963ca7ce5b5d3674626

practicalswift · 2017-11-21T16:58:48Z

Added another commit with two more missing locks:

calling function IsSpent requires holding mutex pwallet->cs_wallet exclusively
writing variable nWalletVersion, nWalletMaxVersion, nOrderPosNext and nTimeFirstKey require holding mutex cs_wallet

practicalswift · 2017-11-21T16:59:12Z

@promag Would you mind re-reviewing? :-)

TheBlueMatt · 2017-12-12T22:38:14Z

src/wallet/wallet.cpp

-        while (pindexRescan && walletInstance->nTimeFirstKey && (pindexRescan->GetBlockTime() < (walletInstance->nTimeFirstKey - TIMESTAMP_WINDOW))) {
-            pindexRescan = chainActive.Next(pindexRescan);
+        {
+            LOCK(walletInstance->cs_wallet);


Given this is effectively just the wallet's constructor, might as well just take the lock further up and hold it the whole time.

@TheBlueMatt Moving it further up will make the scope of the walletInstance->cs_wallet lock cover the ScanForWalletTransactions call. ScanForWalletTransactions locks cs_main giving us the lock order:

cs_wallet

cs_main

This is a potential deadlock given that the opposite lock order is used extensively:

$ git grep "LOCK2(cs_main,.*cs_wallet);" | wc -l 89

Please advice :-)

A solution to that is to also lock cs_main.. but meh

TheBlueMatt · 2017-12-13T19:43:39Z

src/wallet/wallet.h

-        nWalletVersion = FEATURE_BASE;
-        nWalletMaxVersion = FEATURE_BASE;
+        {
+            LOCK(cs_wallet);


Same here. Just add a cs_wallet lock for the whole function.

practicalswift · 2018-02-22T20:25:50Z

@TheBlueMatt Thanks for reviewing! Feedback addressed. Please re-review :-)

practicalswift · 2018-03-02T16:58:30Z

Fixed build issue. Please re-review :-)

maflcko · 2018-03-02T17:05:47Z

Given that https://github.com/bitcoin/bitcoin/pull/11226/files#diff-12635a58447c65585f51d32b7e04075bR857 is now closed, wouldn't it make sense to add the clang annotations within this commit?

practicalswift · 2018-03-10T13:27:17Z

@MarcoFalke @TheBlueMatt @promag Thanks for reviewing. I've now addressed the feedback and added corresponding GUARDED_BY/EXCLUSIVE_LOCKS_REQUIRED annotations. Please re-review :-)

maflcko · 2018-03-11T22:12:31Z

src/wallet/wallet.cpp

@@ -3930,6 +3938,7 @@ CWallet* CWallet::CreateWalletFromFile(const std::string& name, const fs::path&
    int64_t nStart = GetTimeMillis();
    bool fFirstRun = true;
    CWallet *walletInstance = new CWallet(name, CWalletDBWrapper::Create(path));
+    LOCK(walletInstance->cs_wallet);
    DBErrors nLoadWalletRet = walletInstance->LoadWallet(fFirstRun);


Note that LoadWallet takes LOCK2(cs_main, cs_wallet), so taking cs_wallet before cs_main leads to a deadlock in case another thread takes cs_main and then cs_wallet, no?

Thanks. Now fixed. See #11634 (comment).

practicalswift · 2018-03-12T16:02:05Z

Please re-review :-)

practicalswift · 2018-10-09T10:19:36Z

Updated!

Added GUARDED_BY(cs_wallet) annotations also for encrypted_batch, nWalletMaxVersion, m_max_keypool_index and nOrderPosNext.

Rationale:

AddKeyPubKeyWithDB(...) reads encrypted_batch which potentially races with write in the same method
IncOrderPosNext(...) reads nOrderPosNext which potentially races with write in BlockDisconnected(...)
LoadKeyPool(...) reads m_max_keypool_index which potentially races with write in BlockDisconnected(...)
LoadMinVersion(...) reads nWalletMaxVersion which potentially races with write in BlockDisconnected(...)

Please review :-)

practicalswift · 2018-10-09T12:00:30Z

Added GUARDED_BY(cs_wallet) for setExternalKeyPool, mapKeyMetadata, m_script_metadata and setLockedCoins.

Rationale:

$ git grep -E 'AssertLockHeld.*(setExternalKeyPool|mapKeyMetadata|m_script_metadata|setLockedCoins)' | sort | uniq -c
      4 src/wallet/wallet.cpp:    AssertLockHeld(cs_wallet); // mapKeyMetadata
      1 src/wallet/wallet.cpp:    AssertLockHeld(cs_wallet); // m_script_metadata
      1 src/wallet/wallet.cpp:    AssertLockHeld(cs_wallet); // setExternalKeyPool
      5 src/wallet/wallet.cpp:    AssertLockHeld(cs_wallet); // setLockedCoins

…cript_metadata and setLockedCoins

practicalswift · 2018-10-24T08:35:59Z

@MarcoFalke @promag @TheBlueMatt Would you mind reviewing the updated version? :-)

maflcko · 2018-10-24T08:52:19Z

utACK 69e7ee2

…to wallet 69e7ee2 Add GUARDED_BY(cs_wallet) for setExternalKeyPool, mapKeyMetadata, m_script_metadata and setLockedCoins (practicalswift) 37b2538 Add GUARDED_BY(cs_wallet) for encrypted_batch, nWalletMaxVersion, m_max_keypool_index and nOrderPosNext (practicalswift) dee4292 wallet: Add Clang thread safety analysis annotations (practicalswift) 1c7e25d wallet: Add missing locks (practicalswift) Pull request description: Add missing wallet locks: * Calling the function `GetConflicts(...)` requires holding the mutex `cs_wallet` * Calling the function `IsSpent(...)` requires holding the mutex `cs_wallet` * Accessing the variables `mapKeys` and `mapCryptedKeys` requires holding the mutex `cs_KeyStore` * Accessing the variable `nTimeFirstKey` requires holding the mutex `cs_wallet` * Accessing the variable `mapWallet` requires holding the mutex `cs_wallet` * Accessing the variable `nTimeFirstKey` requires holding the mutex `cs_wallet` Tree-SHA512: 8a7b9a4e1f2147e77c04b817617a06304a2e2159148d3eb3514a3c09c41d77ef7e773df6e63880ad9acc026e00690f72d0c51f3f86279177f672d477423accca

promag

utACK 69e7ee2.

promag · 2018-10-24T09:03:43Z

src/wallet/wallet.cpp

@@ -4093,7 +4093,7 @@ std::shared_ptr<CWallet> CWallet::CreateWalletFromFile(const std::string& name,
    // Try to top up keypool. No-op if the wallet is locked.
    walletInstance->TopUpKeyPool();

-    LOCK(cs_main);
+    LOCK2(cs_main, walletInstance->cs_wallet);


No annotation requires this lock right?

Both cs_main and walletInstance->cs_wallet are required from what I can tell.

Removing that lock results in the following:

wallet/wallet.cpp:4104:28: error: calling function 'FindForkInGlobalIndex' requires holding mutex 'cs_main' exclusively [-Werror,-Wthread-safety-analysis] pindexRescan = FindForkInGlobalIndex(chainActive, locator); ^ wallet/wallet.cpp:4131:48: error: reading variable 'nTimeFirstKey' requires holding mutex 'walletInstance->cs_wallet' [-Werror,-Wthread-safety-analysis] while (pindexRescan && walletInstance->nTimeFirstKey && (pindexRescan->GetBlockTime() < (walletInstance->nTimeFirstKey - TIMESTAMP_WINDOW))) { ^ wallet/wallet.cpp:4131:114: error: reading variable 'nTimeFirstKey' requires holding mutex 'walletInstance->cs_wallet' [-Werror,-Wthread-safety-analysis] while (pindexRescan && walletInstance->nTimeFirstKey && (pindexRescan->GetBlockTime() < (walletInstance->nTimeFirstKey - TIMESTAMP_WINDOW))) { ^ wallet/wallet.cpp:4156:77: error: reading variable 'mapWallet' requires holding mutex 'walletInstance->cs_wallet' [-Werror,-Wthread-safety-analysis] std::map<uint256, CWalletTx>::iterator mi = walletInstance->mapWallet.find(hash); ^ wallet/wallet.cpp:4157:43: error: reading variable 'mapWallet' requires holding mutex 'walletInstance->cs_wallet' [-Werror,-Wthread-safety-analysis] if (mi != walletInstance->mapWallet.end()) ^ wallet/wallet.cpp:4181:90: error: calling function 'GetKeyPoolSize' requires holding mutex 'walletInstance->cs_wallet' exclusively [-Werror,-Wthread-safety-analysis] walletInstance->WalletLogPrintf("setKeyPool.size() = %u\n", walletInstance->GetKeyPoolSize()); ^ wallet/wallet.cpp:4182:90: error: reading variable 'mapWallet' requires holding mutex 'walletInstance->cs_wallet' [-Werror,-Wthread-safety-analysis] walletInstance->WalletLogPrintf("mapWallet.size() = %u\n", walletInstance->mapWallet.size()); ^

Right, the new annotations..

It could make sense to avoid calling uiInterface.LoadWallet() with the lock held.

@promag Ah, now I follow. Suggested fix below. What do you think?

diff --git a/src/wallet/wallet.cpp b/src/wallet/wallet.cpp index 29014790e..2deeb9c42 100644 --- a/src/wallet/wallet.cpp +++ b/src/wallet/wallet.cpp @@ -4093,7 +4093,7 @@ std::shared_ptr<CWallet> CWallet::CreateWalletFromFile(const std::string& name, // Try to top up keypool. No-op if the wallet is locked. walletInstance->TopUpKeyPool(); - LOCK2(cs_main, walletInstance->cs_wallet); + LOCK(cs_main); CBlockIndex *pindexRescan = chainActive.Genesis(); if (!gArgs.GetBoolArg("-rescan", false)) @@ -4128,6 +4128,7 @@ std::shared_ptr<CWallet> CWallet::CreateWalletFromFile(const std::string& name, // No need to read and scan block if block was created before // our wallet birthday (as adjusted for block time variability) + LOCK(walletInstance->cs_wallet); while (pindexRescan && walletInstance->nTimeFirstKey && (pindexRescan->GetBlockTime() < (walletInstance->nTimeFirstKey - TIMESTAMP_WINDOW))) { pindexRescan = chainActive.Next(pindexRescan); } @@ -4178,6 +4179,7 @@ std::shared_ptr<CWallet> CWallet::CreateWalletFromFile(const std::string& name, walletInstance->SetBroadcastTransactions(gArgs.GetBoolArg("-walletbroadcast", DEFAULT_WALLETBROADCAST)); { + LOCK(walletInstance->cs_wallet); walletInstance->WalletLogPrintf("setKeyPool.size() = %u\n", walletInstance->GetKeyPoolSize()); walletInstance->WalletLogPrintf("mapWallet.size() = %u\n", walletInstance->mapWallet.size()); walletInstance->WalletLogPrintf("mapAddressBook.size() = %u\n", walletInstance->mapAddressBook.size());

I think that currently there is no harm in calling uiInterface.LoadWallet() with the lock held but if it's not necessary then I wouldn't change that. The diff looks ok.

zcash: cherry picked from commit 1c7e25d zcash: bitcoin/bitcoin#11634

zcash: cherry picked from commit dee4292 zcash: bitcoin/bitcoin#11634

…ax_keypool_index and nOrderPosNext * AddKeyPubKeyWithDB(...) reads encrypted_batch which potentially races with write in the same method. * IncOrderPosNext(...) reads nOrderPosNext which potentially races with write in BlockDisconnected(...). * LoadKeyPool(...) reads m_max_keypool_index which potentially races with write in BlockDisconnected(...). * LoadMinVersion(...) reads nWalletMaxVersion which potentially races with write in BlockDisconnected(...). zcash: only nWalletMaxVersion, nOrderPosNext zcash: cherry picked from commit 37b2538 zcash: bitcoin/bitcoin#11634

…cript_metadata and setLockedCoins zcash: cherry picked from commit 69e7ee2 zcash: bitcoin/bitcoin#11634

zcash: cherry picked from commit 1c7e25d zcash: bitcoin/bitcoin#11634

zcash: cherry picked from commit dee4292 zcash: bitcoin/bitcoin#11634

…ax_keypool_index and nOrderPosNext * AddKeyPubKeyWithDB(...) reads encrypted_batch which potentially races with write in the same method. * IncOrderPosNext(...) reads nOrderPosNext which potentially races with write in BlockDisconnected(...). * LoadKeyPool(...) reads m_max_keypool_index which potentially races with write in BlockDisconnected(...). * LoadMinVersion(...) reads nWalletMaxVersion which potentially races with write in BlockDisconnected(...). zcash: only nWalletMaxVersion, nOrderPosNext zcash: cherry picked from commit 37b2538 zcash: bitcoin/bitcoin#11634

…cript_metadata and setLockedCoins zcash: cherry picked from commit 69e7ee2 zcash: bitcoin/bitcoin#11634

…to wallet 69e7ee2 Add GUARDED_BY(cs_wallet) for setExternalKeyPool, mapKeyMetadata, m_script_metadata and setLockedCoins (practicalswift) 37b2538 Add GUARDED_BY(cs_wallet) for encrypted_batch, nWalletMaxVersion, m_max_keypool_index and nOrderPosNext (practicalswift) dee4292 wallet: Add Clang thread safety analysis annotations (practicalswift) 1c7e25d wallet: Add missing locks (practicalswift) Pull request description: Add missing wallet locks: * Calling the function `GetConflicts(...)` requires holding the mutex `cs_wallet` * Calling the function `IsSpent(...)` requires holding the mutex `cs_wallet` * Accessing the variables `mapKeys` and `mapCryptedKeys` requires holding the mutex `cs_KeyStore` * Accessing the variable `nTimeFirstKey` requires holding the mutex `cs_wallet` * Accessing the variable `mapWallet` requires holding the mutex `cs_wallet` * Accessing the variable `nTimeFirstKey` requires holding the mutex `cs_wallet` Tree-SHA512: 8a7b9a4e1f2147e77c04b817617a06304a2e2159148d3eb3514a3c09c41d77ef7e773df6e63880ad9acc026e00690f72d0c51f3f86279177f672d477423accca

fanquake added the Wallet label Nov 8, 2017

practicalswift force-pushed the missing-wallet-locks branch from 865b5d5 to b229a8e Compare November 8, 2017 09:38

promag reviewed Nov 8, 2017

View reviewed changes

practicalswift force-pushed the missing-wallet-locks branch from b229a8e to e921615 Compare November 8, 2017 13:50

promag reviewed Nov 8, 2017

View reviewed changes

practicalswift force-pushed the missing-wallet-locks branch from e921615 to 007fcbf Compare November 8, 2017 16:01

ryanofsky mentioned this pull request Nov 10, 2017

[WIP] Add Clang thread safety analysis annotations: GUARDED_BY(lock) / EXCLUSIVE_LOCKS_REQUIRED(lock) #11226

Closed

practicalswift force-pushed the missing-wallet-locks branch from 007fcbf to fba0830 Compare November 21, 2017 16:57

TheBlueMatt reviewed Dec 13, 2017

View reviewed changes

practicalswift force-pushed the missing-wallet-locks branch 2 times, most recently from 8d97924 to fca14f6 Compare February 22, 2018 20:25

practicalswift force-pushed the missing-wallet-locks branch from fca14f6 to 8e64007 Compare March 2, 2018 08:19

practicalswift force-pushed the missing-wallet-locks branch 4 times, most recently from 0af5880 to 491ec75 Compare March 10, 2018 13:24

maflcko reviewed Mar 11, 2018

View reviewed changes

practicalswift force-pushed the missing-wallet-locks branch 2 times, most recently from f3cb135 to e7eaf64 Compare March 12, 2018 15:24

practicalswift force-pushed the missing-wallet-locks branch from e7eaf64 to 0f8d1eb Compare March 14, 2018 19:08

practicalswift force-pushed the missing-wallet-locks branch from 0bf4395 to 37b2538 Compare October 9, 2018 10:17

Add GUARDED_BY(cs_wallet) for setExternalKeyPool, mapKeyMetadata, m_s…

69e7ee2

…cript_metadata and setLockedCoins

practicalswift force-pushed the missing-wallet-locks branch from 3431464 to 69e7ee2 Compare October 9, 2018 13:04

DrahtBot mentioned this pull request Oct 20, 2018

rpcwallet: listsentbyaddress RPC #14498

Closed

maflcko merged commit 69e7ee2 into bitcoin:master Oct 24, 2018

promag reviewed Oct 24, 2018

View reviewed changes

practicalswift mentioned this pull request Oct 24, 2018

wallet: Avoid calling uiInterface.LoadWallet(...) with cs_wallet held #14560

Closed

LarryRuane mentioned this pull request Apr 5, 2021

Bitcoin 0.18 locking PRs zcash/zcash#5050

Open

practicalswift deleted the missing-wallet-locks branch April 10, 2021 19:36

LarryRuane pushed a commit to LarryRuane/zcash that referenced this pull request Apr 29, 2021

wallet: Add missing locks

b2cdcd0

zcash: cherry picked from commit 1c7e25d zcash: bitcoin/bitcoin#11634

LarryRuane pushed a commit to LarryRuane/zcash that referenced this pull request Apr 29, 2021

wallet: Add Clang thread safety analysis annotations

a0df9ac

zcash: cherry picked from commit dee4292 zcash: bitcoin/bitcoin#11634

LarryRuane pushed a commit to LarryRuane/zcash that referenced this pull request Apr 29, 2021

Add GUARDED_BY(cs_wallet) for setExternalKeyPool, mapKeyMetadata, m_s…

973d009

…cript_metadata and setLockedCoins zcash: cherry picked from commit 69e7ee2 zcash: bitcoin/bitcoin#11634

LarryRuane pushed a commit to LarryRuane/zcash that referenced this pull request Jun 1, 2021

wallet: Add missing locks

20ce9cc

zcash: cherry picked from commit 1c7e25d zcash: bitcoin/bitcoin#11634

LarryRuane pushed a commit to LarryRuane/zcash that referenced this pull request Jun 1, 2021

wallet: Add Clang thread safety analysis annotations

12a307c

zcash: cherry picked from commit dee4292 zcash: bitcoin/bitcoin#11634

LarryRuane pushed a commit to LarryRuane/zcash that referenced this pull request Jun 1, 2021

Add GUARDED_BY(cs_wallet) for setExternalKeyPool, mapKeyMetadata, m_s…

4bae811

…cript_metadata and setLockedCoins zcash: cherry picked from commit 69e7ee2 zcash: bitcoin/bitcoin#11634

ryanofsky mentioned this pull request Nov 24, 2021

wallet: Call load handlers without cs_wallet locked #22868

Merged

bitcoin locked and limited conversation to collaborators Nov 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wallet: Add missing cs_wallet/cs_KeyStore locks to wallet #11634

wallet: Add missing cs_wallet/cs_KeyStore locks to wallet #11634

practicalswift commented Nov 8, 2017 •

edited

Loading

promag Nov 8, 2017

practicalswift commented Nov 8, 2017

promag Nov 8, 2017

practicalswift Nov 8, 2017 •

edited

Loading

promag Nov 8, 2017

practicalswift Nov 8, 2017

promag commented Nov 8, 2017

practicalswift commented Nov 21, 2017 •

edited

Loading

practicalswift commented Nov 21, 2017

TheBlueMatt Dec 12, 2017

practicalswift Mar 12, 2018 •

edited

Loading

promag Mar 14, 2018

TheBlueMatt Dec 13, 2017

practicalswift commented Feb 22, 2018

practicalswift commented Mar 2, 2018

maflcko commented Mar 2, 2018

practicalswift commented Mar 10, 2018

maflcko Mar 11, 2018

practicalswift Mar 12, 2018

practicalswift commented Mar 12, 2018

practicalswift commented Oct 9, 2018

practicalswift commented Oct 9, 2018

practicalswift commented Oct 24, 2018 •

edited

Loading

maflcko commented Oct 24, 2018

promag left a comment

promag Oct 24, 2018

practicalswift Oct 24, 2018 •

edited

Loading

promag Oct 24, 2018 •

edited

Loading

practicalswift Oct 24, 2018

promag Oct 24, 2018 •

edited

Loading

wallet: Add missing cs_wallet/cs_KeyStore locks to wallet #11634

wallet: Add missing cs_wallet/cs_KeyStore locks to wallet #11634

Conversation

practicalswift commented Nov 8, 2017 • edited Loading

Choose a reason for hiding this comment

practicalswift commented Nov 8, 2017

Choose a reason for hiding this comment

practicalswift Nov 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

promag commented Nov 8, 2017

practicalswift commented Nov 21, 2017 • edited Loading

practicalswift commented Nov 21, 2017

Choose a reason for hiding this comment

practicalswift Mar 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

practicalswift commented Feb 22, 2018

practicalswift commented Mar 2, 2018

maflcko commented Mar 2, 2018

practicalswift commented Mar 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

practicalswift commented Mar 12, 2018

practicalswift commented Oct 9, 2018

practicalswift commented Oct 9, 2018

practicalswift commented Oct 24, 2018 • edited Loading

maflcko commented Oct 24, 2018

promag left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

practicalswift Oct 24, 2018 • edited Loading

Choose a reason for hiding this comment

promag Oct 24, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

promag Oct 24, 2018 • edited Loading

Choose a reason for hiding this comment

practicalswift commented Nov 8, 2017 •

edited

Loading

practicalswift Nov 8, 2017 •

edited

Loading

practicalswift commented Nov 21, 2017 •

edited

Loading

practicalswift Mar 12, 2018 •

edited

Loading

practicalswift commented Oct 24, 2018 •

edited

Loading

practicalswift Oct 24, 2018 •

edited

Loading

promag Oct 24, 2018 •

edited

Loading

promag Oct 24, 2018 •

edited

Loading