Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ChainstateManager, remove BlockManager global #17737

Merged
merged 6 commits into from Apr 10, 2020

Conversation

jamesob
Copy link
Member

@jamesob jamesob commented Dec 12, 2019

This is part of the assumeutxo project:

Parent PR: #15606
Issue: #15605
Specification: https://github.com/jamesob/assumeutxo-docs/tree/master/proposal


This changeset introduces ChainstateManager, which is responsible for creating and managing access to multiple chainstates. Until we allow chainstate creation from UTXO snapshots (next assumeutxo PR?) it's basically unnecessary, but it is a prerequisite for background IBD support.

Changes are also made to the initialization process to make use of g_chainman and thus clear the way for multiple chainstates being loaded on startup.

One immediate benefit of this change is that we no longer have the g_blockman global, but instead have the ChainstateManager inject a reference of its shared BlockManager into any chainstate it creates.

Another immediate benefit is that uses of ChainActive() and ChainstateActive() are now covered by lock annotations. Because use of g_chainman is annotated to require cs_main, these two functions subsequently follow.

Because of whitespace changes, this diff looks bigger than it is. E.g., 4813167 is most easily reviewed with

git show --color-moved=dimmed_zebra -w 4813167d98

@fanquake fanquake added this to In progress in Assume UTXO Dec 12, 2019
@DrahtBot
Copy link
Contributor

DrahtBot commented Dec 12, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@jamesob jamesob force-pushed the 2019-12-au.chainman branch 2 times, most recently from ce74d24 to 2328317 Compare December 13, 2019 15:12
Copy link
Member

@MarcoFalke MarcoFalke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 0226408, did not read the tests 🔲

Show signature and timestamp

Signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

ACK 0226408999, did not read the tests 🔲
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYEACgkQzit1aX5p
pUixgQv9ELRmdepH+VtXfc0mvkvHjZZAGrvdNEAGLqlEK+nbDy0YtUnZtECto3aJ
K/fFg7heGCu3/pRgx25GGDnaRNrOPc+q56kWXUjSS7lweiyx1JYw4V8sVlgas3Ih
4Cxbgy3GxmPL88e1mRBpqBdyKxGi3cPxpPjew63/sllo9ZDPBMsA5g0NcpUwIC2H
TMgdLJbffnHCgCKkIH5/lXah8W7fF6Tt/wyzK4RaijwcxjVrdFheJTv0kLiBJDAG
AmHMVhKDQTBeZFn5CydqeupqV90u7rdFd/S6MliuawQGwtlobeqKqs7jxeR23+Yx
AfAAk6nKLhQFQ5x998j/rGamIcYO7RCRE0be0dZC1KGdhTwIY1LOsSKHlJAn64Ik
XKNJXDbg8o1vnBgAEmzslunwooaOoF2oSyEB4crt++JNYLl38imrOdZ+0kdlQ/BC
GCRJWD/y2WSNRxCrkyJZ6uxTkUf7KmEbh8qwOavXby2DR2rOSOGE9nM9ewXgqbsK
BDO3176X
=Ygdp
-----END PGP SIGNATURE-----

Timestamp of file with hash c4bdc7880969f1e6719cf25c6f715a20e103935d1147395578a097e3179189ff -

src/validation.cpp Show resolved Hide resolved
src/validation.cpp Show resolved Hide resolved
src/validation.h Outdated Show resolved Hide resolved
src/validation.h Outdated Show resolved Hide resolved
src/validation.cpp Outdated Show resolved Hide resolved
src/validation.h Show resolved Hide resolved
src/validation.h Show resolved Hide resolved
src/validation.cpp Show resolved Hide resolved
src/validation.h Outdated Show resolved Hide resolved
src/init.cpp Show resolved Hide resolved
Copy link
Member

@ariard ariard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 6d14916 to 4813167.

Need to dig into parent PR a bit more before to review forward.

src/validation.h Outdated Show resolved Hide resolved
src/validation.h Outdated Show resolved Hide resolved
src/validation.cpp Outdated Show resolved Hide resolved
src/validation.h Outdated Show resolved Hide resolved
src/validation.h Show resolved Hide resolved
src/validation.h Outdated Show resolved Hide resolved
src/validation.cpp Show resolved Hide resolved
src/init.cpp Outdated Show resolved Hide resolved
src/init.cpp Show resolved Hide resolved
@jonatack
Copy link
Member

Bringing in this IRC comment, if helpful:

"recent utxo snapshots for testing are available here: #15606 (comment)"

src/validation.cpp Show resolved Hide resolved
src/validation.cpp Show resolved Hide resolved
src/validation.h Outdated Show resolved Hide resolved
@MarcoFalke
Copy link
Member

ACK 16d20d9 only, will continue review later on 🚀

Show signature and timestamp

Signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

ACK 16d20d91b43b1c30333fe14272d69cfb543229b3 only, will continue review later on 🚀
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUjn6wv/cGwiv/yWquRmx2FEjuO62k8716JXtWFhx7zE1OcbgSYtOx2i4QAPQJhJ
FYhRtUuVArBrkz9lrn9VSCo3Aueb7w8b/j3/8dGyH/zqUU7IhWgrjC7qlfbdP4lp
DzDGE5ITqlvZlEkYAloI3zMoVRaUHDlGTK+bEHK3WdRnsPL9oR/C+S1UNcPNqq1m
I4G5UrneAHnQnIiWJfjfzpZ0BlXCXCfs3S4BBwKNTQAMlqlgzz1sCQr5R2d8kcW3
mNexZ76vn3IlrmFAyXWNwzQvGvfrGMF4RcIYQiS5OLfcjlzLfZOxSQJjerHzMe7C
EdnUsD+OX6BA6ijgU7eO7k5mdRxSkUVLanAzINUB5dupiw1NscUyZlqsH2J549DT
Q2g6va4jFXAULCFg2Rwx18k3GOQ0spLsRyv6wn/PxpwzEaZnEnSsoCe8pdiwCaOk
SyIJzqx3vzfA3IhEgHVnLHqLN8iYk9F5XXevDDA7fPN1A/KAWDcQXjVEpu3jA03w
nuOXSfZ+
=G8F6
-----END PGP SIGNATURE-----

Timestamp of file with hash 7e94e353f869ccf97b1947f28a0c0193a5e2cf8d984598592f1296d23f14168d -

src/init.cpp Outdated Show resolved Hide resolved
src/init.cpp Outdated Show resolved Hide resolved
Copy link
Member

@MarcoFalke MarcoFalke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK e7bc059 only, have not reviewed later commits closely 🕧

Show signature and timestamp

Signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

ACK e7bc059491d51919f676b35b6843d1c395d452a6 only, have not reviewed later commits closely 🕧
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUjKXwwApeBAJd4ix/yrfHo9bHi30caJGej5Xk3cCvsbTorNmxVgMjJdbcALeADr
AbWHnVntgHiI7oLtzbKrFlKTGi826+stmRIy2VPyVs31L/z7k2pLKJsRFtYO+kzv
ATQ4POaLd58wzFI4bxGtSLBu7I+91f6iLzdtdINYDVJj6g89eBd74ICseFoQjM6x
lAugPzMXzrL8kh71IFBumNpQKq3rNZ9Wb/6oqVCBiZUGUnW4qC3a5aVDhwfnBnKX
YQia7Cej4Ex22+FAxh+4LUl8nO6I+UMH53TASaMG/ZoHUrXpHouWyo9vmUAwBMyp
gqN9AX/V5Hv7YCk3hehviw6p9ZI+Gvs3hrXYlU2FLUFTGXJfQz5yM6jSipnK5tZC
vqx46MazAjQ7x2de/HJIfwf+uEFpZkohauhobu5pMGss0tuVSiWVcofIoS8I5zei
mlEZ6atiGucuEMZmiC8+z3g4TbGo5kztOB00YOVVbY1+jDmuOOs+T220p4FtrDTl
B6J3CRmr
=Ik+8
-----END PGP SIGNATURE-----

Timestamp of file with hash fe215ba014f43a48217fc4a11afdd4fb671d3ccd7e48f8bed366d49affae1228 -

src/init.cpp Show resolved Hide resolved
src/init.cpp Outdated Show resolved Hide resolved
src/net_processing.cpp Outdated Show resolved Hide resolved
@MarcoFalke
Copy link
Member

ACK 1c17381, but it looks like the tests fail 🍢

Show signature and timestamp

Signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

ACK 1c173812bac34e263ae49e0bc80e45b437e95e8b, but it looks like the tests fail 🍢
-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUjFNgwApRAQI5N2D4113l6z1rLjgUjlixBjaHFnZC0rG5naAL/Gu55ofeAmzg5Z
TYF1WvqL9spMC+CDCweOyRGKsWb81Ryv8Ieni+Dn8RE+pu0VLakDu9Lw95+bd43j
1P6RLJYa2NYFlCuJssbs5ziEPJqs3/DlelgWPslfbdgwLXWFT54CvijXuFEF43b/
T9/NBr1OO1ByA0+AEM0eT3iHH55Guna544fuY8UjEDabnjqs8EiatmXArGlqi6Wr
D6U7J33USdtpof1bb+2zR8KLSb4MCFFxezr2yrSREHf5tqqamfN3qJ2dYHazNEMd
S4ka7rOrQqY/c2XIBs4oYD8pxI3thvxuo4nAnsG0sxjxkW0mm4O5OBqTFP/aGilX
NNkR1TTl1DcuqbTFK3FNoiqmG/yuh2OWw3jm0EG6jhEcf3sdf1d2e5RUQ0xKPDRm
h/PVqKPlmRDkEHiT9XoLgWDw6C9fH86HdGh0HLBZmTJ2jOt+sVeIvSXtth7sVznG
vtfM49Wg
=0mLY
-----END PGP SIGNATURE-----

Timestamp of file with hash 6a38ed3fc7e109b015b25d8cd883d76b0d7c7f56f8c96dd80e10381889e51bbd -

Copy link
Contributor

@ryanofsky ryanofsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review ACK 1c17381. I did have a bunch of questions and suggestions below, but the suggestions aren't important, and if the questions don't point to real problems, I think this looks good to merge as is.

src/validation.cpp Show resolved Hide resolved
src/validation.h Outdated Show resolved Hide resolved
src/validation.h Outdated
* *Background validation chainstate*: an IBD chainstate for which the
* IBD process is happening in the background while use of the
* active chainstate allows the rest of the system to function.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "validation: introduce unused ChainstateManager" (10837ab)

The "Background validation chainstate" term seems a little awkward to me, where I guess every background validation chainstate is an IBD chainstate, but an IBD chainstate isn't a background validation chainstate until a separate snapshot chainstate has been loaded.

It seems like if you replaced the IsBackgroundValidationChainstate() method with plainer ChainStateManger::IsIBD(chainstate) and ChainStateManager::HasSnapshot() methods you could drop the "background validation chainstate" term and method and make things easier to follow.

Or if having the term is more useful, maybe call it a "background IBD chainstate" instead of "background validation chainstate" to be more obvious it's always refers to the IBD chain.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What gets kind of confusing here is that chainstates created from snapshots still technically go through IBD, since by the time the snapshot is actually used to create a chainstate, it's far enough behind the tip that it technically undergoes IBD (albeit an abbreviated one). So I think referring to what I now call the background validation chainstate as the IBD chainstate might be confusing in that sense.

But I agree with you that the naming here isn't ideal. Definitely open to changing this if you have other good ideas.

Copy link
Contributor

@ryanofsky ryanofsky Jan 28, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #17737 (comment)

I guess I see that but it would seem more like a reason to avoid the term "IBD chainstate" entirely.

Since chain state manager only has a snapshot chainstate and a non snapshot chainstate, I like the idea of just referring to the two chainstates with the same names consistently, and not muddying the water with other chainstate names and definitions. "IBD chainstate" is what you call the non-snapshot chainstate everywhere except in this one method and comment, which is my why first suggestion is delete "background validation chainstate" comment and have separate methods replacing IsBackgroundValidationChainstate. Switching to "Background IBD chainstate" was just a runner up suggestion.

I guess also I don't find the "IBD chainstate" name to be too confusing because I take IBD chainstate to mean chainstate generated entirely from IBD, not a state with some UTXOs that happened to come from IBD. But for another naming suggestion, maybe an alternative would be to use "snapshot chainstate" and "blocksonly chainstate" instead of "snapshot chainstate" and "IBD chainstate"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially (?) fixed by changing IsBackgroundValidationChainstate() to IsBackgroundIBD(). I think blocksonly is somewhat confusing because there'd be a confusing overlap with the P2P definition of blocksonly, which I think is different or unrelated.

src/validation.h Outdated Show resolved Hide resolved
src/init.cpp Outdated Show resolved Hide resolved
BOOST_CHECK(c2.ActivateBestChain(_, chainparams, nullptr));

BOOST_CHECK(manager.IsSnapshotActive());
BOOST_CHECK(!manager.IsSnapshotValidated());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "test: add basic tests for ChainstateManager" (997bf7f)

No test for the case where IsSnapshotValidated returns true, I guess. Understandable if it would make the test setup too complicated

src/init.cpp Outdated Show resolved Hide resolved
src/init.cpp Show resolved Hide resolved
src/init.cpp Outdated Show resolved Hide resolved
src/validation.h Outdated Show resolved Hide resolved
@jamesob
Copy link
Member Author

jamesob commented Jan 28, 2020

Thanks for the good reviews! I'll be following up shortly with corresponding changes.

I'd previously attempted to create a specialized lock for ChainstateManager,
but it turns out that because that lock would be required for functions like
ChainActive() and ChainstateActive(), it created irreconcilable lock inversions
since those functions are used so broadly throughout the codebase.

Instead, I'm just using cs_main to protect the contents of g_chainman.

Co-authored-by: Russell Yanofsky <russ@yanofsky.org>
Copy link
Contributor

@fjahr fjahr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review ACK abfc152

Had a few nits that can well be ignored or addressed in next PR in the assume-utxo series.

General note: I was not sure about some of the ChainstateManager API, some of the methods seemed like they could be private (IsSnapshotValidated() seems to be only used internally) and others are not used at all (IsSnapshotValidated and SnapshotBlockhash for example) so they could have been included in the next PRs when they are needed. But I can wait and see how they are used in the next PR. :)

@@ -1254,6 +1256,10 @@ void CChainState::InitCoinsDB(
bool should_wipe,
std::string leveldb_name)
{
if (!m_from_snapshot_blockhash.IsNull()) {
leveldb_name += "_" + m_from_snapshot_blockhash.ToString();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Do you anticipate more than one snapshot to be present? From the rest of the code it seems to me that this will not be the case. Then I think naming the db just "snapshot" or so is more intuitive.


to_modify.reset(new CChainState(snapshot_blockhash));

// Snapshot chainstates and initial IBD chaintates always become active.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: chaintates

@@ -698,11 +698,17 @@ static void ThreadImport(std::vector<fs::path> vImportFiles)
}

// scan for better chains in the block chain database, that are not yet connected in the active best chain
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: That comment is still relevant to the code below but because of the empty line it seems detached.

@fjahr
Copy link
Contributor

fjahr commented Mar 17, 2020

Code Review Re-ACK abfc152

@MarcoFalke
Copy link
Member

re-ACK c9017ce 📙

Only changes since my last review:

  • Fix lock order in tests
  • Making the snapshot blockhash optional
Show signature and timestamp

Signature:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

re-ACK c9017ce3bc27665594c9d80f395780d40755bb22 📙

Only changes since my last review:

* Fix lock order in tests
* Making the snapshot blockhash optional

-----BEGIN PGP SIGNATURE-----

iQGzBAEBCgAdFiEE+rVPoUahrI9sLGYTzit1aX5ppUgFAlwqrYAACgkQzit1aX5p
pUjHLQv/RAhLnsTpzdzOpmyvTALXqbxQut9dguj1Tt46DsWnRzIcrwwcsYHbDreU
B3DmuqYHMdcb1o/OWV5IvWiHMnUbY/EbFfD6Zw9/4HNhHxmruCgUrG9+zP+oD1ck
nacI226jyJQWVKEz6XtbuF7uhW1an9J2XyPbnh6EaUywTB+CabLI/mnOFiHerWGd
Q4VwcxSFYeBjmoDP1ekDaTafAphHhDSDO7MaissHBPq+1HliMURndLGgDKpRnT0y
Fn7dfmN/fN3bS5JWufUCaatPdYjES5UOWRaDasub0ipmlvBXzTk/Z48lYzqz4QEp
Owa9fl6EoiRrhsQQSMOdEm6Mi2RCkwIXTK85+Ix7VP7ys+uQov/5fJbxel9zhInK
/4nAtO6uum9XscLqWqHi6AFEVGOEVOfYPN+CJbRW1y5PinpR0t1kRVhEjfXgzWag
bZfFWB2qs9bo+pWMCBDB/xeucJbJC2aDfnDqE5Gqpf1Knrg7t2tVbgU7IWJBaQLb
9KZNfpTL
=hs4D
-----END PGP SIGNATURE-----

Timestamp of file with hash f61dcf850f62e4c3ff5ca63de6c032928f672c45311925c6f66a2497f5bc0c48 -

@MarcoFalke
Copy link
Member

@fjahr It looks like your commit hash is the same in the re-ACK

@fjahr
Copy link
Contributor

fjahr commented Mar 18, 2020

@fjahr It looks like your commit hash is the same in the re-ACK

Ugh, how did I do that? I definitely re-reviewed the new code.

Code Review Re-ACK c9017ce

Copy link
Member

@ariard ariard left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review ACK c9017ce

Side-node: there is a g_chainman access in LoadBlockIndex (src/validation.cpp L4577), not an issue right now because caller already holds cs_main but if function is moved in the future a AssertLockHeld would be better.

assert(g_chainman.m_active_chainstate);
return *g_chainman.m_active_chainstate;
}

CChain& ChainActive()
{
LOCK(::cs_main);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

c9017ce

Is this LOCK really necessary given we call ::ChainstateActive() just after ? Doesn't hurt because we use recursive_mutex but still..

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #17737 (comment)

Is this LOCK really necessary given we call ::ChainstateActive() just after ? Doesn't hurt because we use recursive_mutex but still..

Agree it's definitely not necessary (but harmless)

Copy link
Contributor

@ryanofsky ryanofsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review ACK c9017ce. No changes since last review other than a straight rebase

@@ -894,7 +921,7 @@ class ChainstateManager
void Reset();
};

extern ChainstateManager g_chainman;
extern ChainstateManager g_chainman GUARDED_BY(::cs_main);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #17737 (comment)

If understand well the new lock model, g_chainman content is under cs_main but not CChainState. If yes, why some GetAll access in init.cpp don't use lock for access ?

I don't think this is true. Maybe it was resolved? You can add EXCLUSIVE_LOCKS_REQUIRED(cs_main) to the GetAll declaration, and it will only fail in test code using a local chain state manager not tied to the cs_main global

assert(g_chainman.m_active_chainstate);
return *g_chainman.m_active_chainstate;
}

CChain& ChainActive()
{
LOCK(::cs_main);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #17737 (comment)

Is this LOCK really necessary given we call ::ChainstateActive() just after ? Doesn't hurt because we use recursive_mutex but still..

Agree it's definitely not necessary (but harmless)

@jamesob
Copy link
Member Author

jamesob commented Mar 31, 2020

This tip has accumulated a good number of code ACKs, so I'm hesitant to address nits here. If anyone thinks that's the right move though I'm happy to.

Edit: and thanks for the reviews, everyone!

@fanquake
Copy link
Member

fanquake commented Apr 1, 2020

If anyone thinks that's the right move though I'm happy to.

I think we can merge this after the 0.20 branch off.

@MarcoFalke MarcoFalke merged commit 10358a3 into bitcoin:master Apr 10, 2020
sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Apr 13, 2020
c9017ce protect g_chainman with cs_main (James O'Beirne)
2b081c4 test: add basic tests for ChainstateManager (James O'Beirne)
4ae29f5 use ChainstateManager to initialize chainstate (James O'Beirne)
5b690f0 refactor: move RewindBlockIndex to CChainState (James O'Beirne)
89cdf4d validation: introduce unused ChainstateManager (James O'Beirne)
8e2ecfe validation: add CChainState.m_from_snapshot_blockhash (James O'Beirne)

Pull request description:

  This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11):

  Parent PR: bitcoin#15606
  Issue: bitcoin#15605
  Specification: https://github.com/jamesob/assumeutxo-docs/tree/master/proposal

  ---

  This changeset introduces `ChainstateManager`, which is responsible for creating and managing access to multiple chainstates. Until we allow chainstate creation from UTXO snapshots (next assumeutxo PR?) it's basically unnecessary, but it is a prerequisite for background IBD support.

  Changes are also made to the initialization process to make use of `g_chainman` and thus clear the way for multiple chainstates being loaded on startup.

  One immediate benefit of this change is that we no longer have the `g_blockman` global, but instead have the ChainstateManager inject a reference of its shared BlockManager into any chainstate it creates.

  Another immediate benefit is that uses of `ChainActive()` and `ChainstateActive()` are now covered by lock annotations. Because use of `g_chainman` is annotated to require cs_main, these two functions subsequently follow.

  Because of whitespace changes, this diff looks bigger than it is. E.g., 4813167 is most easily reviewed with
  ```sh
  git show --color-moved=dimmed_zebra -w 4813167
  ```

ACKs for top commit:
  MarcoFalke:
    re-ACK c9017ce 📙
  fjahr:
    Code Review Re-ACK c9017ce
  ariard:
    Code Review ACK c9017ce
  ryanofsky:
    Code review ACK c9017ce. No changes since last review other than a straight rebase

Tree-SHA512: 3f250d0dc95d4bfd70852ef1e39e081a4a9b71a4453f276e6d474c2ae06ad6ae6a32b4173084fe499e1e9af72dd9007f4a8a375c63ce9ac472ffeaada41ab508
@laanwj laanwj removed this from Blockers in High-priority for review Apr 16, 2020
MarcoFalke added a commit that referenced this pull request May 23, 2020
fab6b9d validation: Mark g_chainman DEPRECATED (MarcoFalke)
fa1d97b validation: Make ProcessNewBlock*() members of ChainstateManager (MarcoFalke)
fa24d49 validation: Make PruneOneBlockFile() a member of ChainstateManager (MarcoFalke)
fa84b1c validation: Make LoadBlockIndex() a member of ChainstateManager (MarcoFalke)
fa05fdf net: Pass chainman into PeerLogicValidation (MarcoFalke)
fa7b626 node: Add chainman alias for g_chainman (MarcoFalke)

Pull request description:

  The global `g_chainman` has recently been introduced in #17737. The chainstate manager is primarily needed for the assumeutxo feature, but it can also simplify testing in the future.

  The goal of this pull is to make the global chainstate manager internal to validation, so that all external code does not depend on globals and that unit or fuzz tests can pass in their (potentially mocked) chainstate manager.

  I suggest reviewing the pull request commit-by-commit. It should be relatively straightforward refactoring that does not change behavior at all.

ACKs for top commit:
  ryanofsky:
    Code review ACK fab6b9d. Had to be rebased but still looks good

Tree-SHA512: dcbf114aeef4f8320d466369769f22ce4dd8f46a846870354df176c3de9ff17c64630fbd777e7121d7470d7a8564ed8d37b77168746e8df7489c6877e55d7b4f
@fanquake fanquake moved this from In progress to Done in Assume UTXO May 25, 2020
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Nov 30, 2020
Summary:
```
This parameter is unused, but in future commits will allow
ChainstateManager to differentiate between chainstates created from a
UTXO snapshot from those that weren't.
```

Partial backport (1/5) of core [[bitcoin/bitcoin#17737 | PR17737]]:
bitcoin/bitcoin@8e2ecfe

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, PiRK

Reviewed By: #bitcoin_abc, PiRK

Differential Revision: https://reviews.bitcoinabc.org/D8537
deadalnix pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Nov 30, 2020
Summary:
```
ChainstateManager is responsible for creating and managing multiple
chainstates, and will provide a high-level interface for accessing the
appropriate chainstate based upon a certain use.
```

Partial backport (2/5) of core [[bitcoin/bitcoin#17737 | PR17737]]:
bitcoin/bitcoin@89cdf4d

Depends on D8537.

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, PiRK

Reviewed By: #bitcoin_abc, PiRK

Differential Revision: https://reviews.bitcoinabc.org/D8538
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Nov 30, 2020
Summary:
```
This allows us to easily initialize multiple chainstates on startup in
future commits. It retires the g_chainstate global in lieu of
g_chainman.
```

Partial backport (3/5) of core [[bitcoin/bitcoin#17737 | PR17737]]:
bitcoin/bitcoin@4ae29f5

Includes a fix to `qt/test/apptests.cpp` from the next commit to unbreak
the bitcoin-qt tests:
bitcoin/bitcoin@2b081c4#diff-41e328d9ba5879d2bd026a700743339b5b300b9bee50844795ef9e27f60bdfb8R85

Depends on D8538.

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, deadalnix

Reviewed By: #bitcoin_abc, deadalnix

Subscribers: deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D8539
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Nov 30, 2020
Summary:
Partial backport (4/5) of core [[bitcoin/bitcoin#17737 | PR17737]]:
bitcoin/bitcoin@2b081c4

Depends on D8539.

Test Plan:
  ninja check

Reviewers: #bitcoin_abc, deadalnix

Reviewed By: #bitcoin_abc, deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D8540
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this pull request Nov 30, 2020
Summary:
```
I'd previously attempted to create a specialized lock for
ChainstateManager, but it turns out that because that lock would be
required for functions like ChainActive() and ChainstateActive(), it
created irreconcilable lock inversions since those functions are used so
broadly throughout the codebase.

Instead, I'm just using cs_main to protect the contents of g_chainman.
```

Completes backport (5/5) of core [[bitcoin/bitcoin#17737 | PR17737]]:
bitcoin/bitcoin@c9017ce

The lock change in `wallet_tests.cpp` is not needed due to out of order
backports.

Depends on D8540.

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, deadalnix

Reviewed By: #bitcoin_abc, deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D8541
@jamesob jamesob mentioned this pull request Aug 5, 2021
17 tasks
PhotoshiNakamoto added a commit to PhotonicBitcoin/pBTC-core that referenced this pull request Dec 11, 2021
Bitcoin Core PR: bitcoin/bitcoin#17737

Pull request description:

  This is part of the [assumeutxo project](https://github.com/bitcoin/bitcoin/projects/11):

  Parent PR: bitcoin/bitcoin#15606
  Issue: bitcoin/bitcoin#15605
  Specification: https://github.com/jamesob/assumeutxo-docs/tree/master/proposal

  ---

  This changeset introduces `ChainstateManager`, which is responsible for creating and managing access to multiple chainstates. Until we allow chainstate creation from UTXO snapshots (next assumeutxo PR?) it's basically unnecessary, but it is a prerequisite for background IBD support.

  Changes are also made to the initialization process to make use of `g_chainman` and thus clear the way for multiple chainstates being loaded on startup.

  One immediate benefit of this change is that we no longer have the `g_blockman` global, but instead have the ChainstateManager inject a reference of its shared BlockManager into any chainstate it creates.

  Another immediate benefit is that uses of `ChainActive()` and `ChainstateActive()` are now covered by lock annotations. Because use of `g_chainman` is annotated to require cs_main, these two functions subsequently follow.

  Because of whitespace changes, this diff looks bigger than it is. E.g., bitcoin/bitcoin@4813167 is most easily reviewed with
  ```sh
  git show --color-moved=dimmed_zebra -w bitcoin/bitcoin@4813167
  ```
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
Assume UTXO
  
Done
Development

Successfully merging this pull request may close these issues.

None yet

10 participants