Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync with the main repo #359

Merged
merged 1,220 commits into from
Aug 31, 2023
Merged

Sync with the main repo #359

merged 1,220 commits into from
Aug 31, 2023

Conversation

hebasto
Copy link
Member

@hebasto hebasto commented Aug 29, 2023

Sync with the main repo up to the latest bitcoin/bitcoin@ab42b2e, which includes the recent changes in the CI.

There is no downloadable artifacts support for now. It will be done in a separated PR(s).

Additionally:

  • The code was adjusted to reflect changes from PR27419, PR27491, PR27576 and PR27636.
  • Fixed modernize-use-default-member-init clang-tidy warnings.
  • The ARM task has been temporarily disabled until the issue with the depends cache is resolved.

Guix builds:

e92b8c4c3298165edb1a0e85ee516d52c81af1269405dcbc6520e63069de2363  guix-build-b3261144c892/output/aarch64-linux-gnu/SHA256SUMS.part
939c6c002490d5649bdbfabacd20cd2270b41b20b7b3a254c9fcd5780209900d  guix-build-b3261144c892/output/aarch64-linux-gnu/bitcoin-b3261144c892-aarch64-linux-gnu-debug.tar.gz
b3c1383fb394997378997bdd2933965cf4ecc694143b4703108ff6ecb946696c  guix-build-b3261144c892/output/aarch64-linux-gnu/bitcoin-b3261144c892-aarch64-linux-gnu.tar.gz
f43fedf3af666d35e83b84e63cfe19f315f74f01296982f47d8c159385c3b03c  guix-build-b3261144c892/output/arm-linux-gnueabihf/SHA256SUMS.part
73b89b0487e8eee474a6c9c96ae0e7ad635cccc332fc062eb5d4ff5555356c3e  guix-build-b3261144c892/output/arm-linux-gnueabihf/bitcoin-b3261144c892-arm-linux-gnueabihf-debug.tar.gz
b4518dd9396f316de8d7de5181b8b5d1083e0afa9081625c37117472d2559380  guix-build-b3261144c892/output/arm-linux-gnueabihf/bitcoin-b3261144c892-arm-linux-gnueabihf.tar.gz
0213e754408e2a032cef61a946354656f5b5f755f85aeac1ce4b37f1d22528e6  guix-build-b3261144c892/output/arm64-apple-darwin/SHA256SUMS.part
11bc1be1f53dad337565f3c556dd69abc2d702a31e661359daad6ff89225c794  guix-build-b3261144c892/output/arm64-apple-darwin/bitcoin-b3261144c892-arm64-apple-darwin-unsigned.dmg
558d8e805420c7a348759df6f559ca349953646aa28840efafe5a3d245ea917f  guix-build-b3261144c892/output/arm64-apple-darwin/bitcoin-b3261144c892-arm64-apple-darwin-unsigned.tar.gz
e679ce3f1c80aff11a5eab8890efbd0d396a851875fbd6f93f32eef5cdf06813  guix-build-b3261144c892/output/arm64-apple-darwin/bitcoin-b3261144c892-arm64-apple-darwin.tar.gz
0cb346390dc6620593b1af5b6669ddc3c1a8d2219a51b1697747c5ab24069c27  guix-build-b3261144c892/output/dist-archive/bitcoin-b3261144c892.tar.gz
ac8bd2d58d9d0ebe2da1c8efa2d57bd97c3ef2b2590c758edbc4919808c528c5  guix-build-b3261144c892/output/powerpc64-linux-gnu/SHA256SUMS.part
cdf8252fa8aca6da61ff6926de5c7e2e6560ab046049c84c26ba44823f83236a  guix-build-b3261144c892/output/powerpc64-linux-gnu/bitcoin-b3261144c892-powerpc64-linux-gnu-debug.tar.gz
3b8b5f53d365b5bf962ecd7def9f06b6f13af0e5c9ef69c6d028f1ed772459be  guix-build-b3261144c892/output/powerpc64-linux-gnu/bitcoin-b3261144c892-powerpc64-linux-gnu.tar.gz
b44e688d233dcb46a7d6d0b1d97979335d3cc559d16190cc5cd647add79298d2  guix-build-b3261144c892/output/powerpc64le-linux-gnu/SHA256SUMS.part
ae5c19afefd523cdc171a3f9aa9f707870fd99749c01c01166086619dfd95ece  guix-build-b3261144c892/output/powerpc64le-linux-gnu/bitcoin-b3261144c892-powerpc64le-linux-gnu-debug.tar.gz
bb581b1444fa1686f8889248af13d1859f2915091cd640bc522185d5ad83e13d  guix-build-b3261144c892/output/powerpc64le-linux-gnu/bitcoin-b3261144c892-powerpc64le-linux-gnu.tar.gz
bdca0a3c19b5a9a5c72b2b43b07050678d960009d3fa80cf7e0689d508346974  guix-build-b3261144c892/output/riscv64-linux-gnu/SHA256SUMS.part
b0b9c91abe2ad0b5ab3b0bfd10c90133d8d75b50aef0a6a98ac2c2ae4219eaa8  guix-build-b3261144c892/output/riscv64-linux-gnu/bitcoin-b3261144c892-riscv64-linux-gnu-debug.tar.gz
fcce0ea00f1d9df136dd677cbc468183faa92bd4bfcd4a77cd1c70f1b894b5f0  guix-build-b3261144c892/output/riscv64-linux-gnu/bitcoin-b3261144c892-riscv64-linux-gnu.tar.gz
7be84969950bb9570522be5a37551c01698cd3fb65eca3988fc9bd6867460552  guix-build-b3261144c892/output/x86_64-apple-darwin/SHA256SUMS.part
25203f50aa6a344ad1c6c4a44a48082440bb0af9bf38f0d60506569f216d1672  guix-build-b3261144c892/output/x86_64-apple-darwin/bitcoin-b3261144c892-x86_64-apple-darwin-unsigned.dmg
16c5baaf6d00ed43b0611c86c2d4555d500b3896daa1daac6a567bc2611c39f6  guix-build-b3261144c892/output/x86_64-apple-darwin/bitcoin-b3261144c892-x86_64-apple-darwin-unsigned.tar.gz
86662f39c29b013b576e6555ecb6cbbc98eaa08532a541e22a7ed6b1baf87209  guix-build-b3261144c892/output/x86_64-apple-darwin/bitcoin-b3261144c892-x86_64-apple-darwin.tar.gz
fbbc0ad2376431fdc5b214fd63f24a6da907d87f6f11e0833def50c0d45772cd  guix-build-b3261144c892/output/x86_64-linux-gnu/SHA256SUMS.part
cba8d700f746a6063809570e45d6dc3d5e60ad5f1a28e0f41f8beed8b546a7b1  guix-build-b3261144c892/output/x86_64-linux-gnu/bitcoin-b3261144c892-x86_64-linux-gnu-debug.tar.gz
0a32985a1e26e13ce883a85e4a92cc68bf51ce096f2f6d74ea499a9fa662d7d0  guix-build-b3261144c892/output/x86_64-linux-gnu/bitcoin-b3261144c892-x86_64-linux-gnu.tar.gz
0bd4cc64cd6ad733cdef87cd74d5034e79dd250b72795cebf9c2c63500509457  guix-build-b3261144c892/output/x86_64-w64-mingw32/SHA256SUMS.part
6ed8f2e6c6cf1992d156672707cd2c254754051f88223dd052a9cd9078d84789  guix-build-b3261144c892/output/x86_64-w64-mingw32/bitcoin-b3261144c892-win64-debug.zip
1ea6d7660652e20b2b1529e406be1f606745d35f6a179b006335a19a19aa9a5b  guix-build-b3261144c892/output/x86_64-w64-mingw32/bitcoin-b3261144c892-win64-setup-unsigned.exe
41b0f8cbac614e8c555921de60b25a73a75e6bed025de98ca40d3db48c5db6b1  guix-build-b3261144c892/output/x86_64-w64-mingw32/bitcoin-b3261144c892-win64-unsigned.tar.gz
5c68d711782e76f9e4be93b5468c505f022b72ca299532b200e58fe1e51343b1  guix-build-b3261144c892/output/x86_64-w64-mingw32/bitcoin-b3261144c892-win64.zip

ajtowns and others added 30 commits August 3, 2023 13:42
…eMany()

Instead of recursively calling `SerializeMany` and peeling off one
argument at a time, use a fold expression. This simplifies the code,
makes it most likely faster because it reduces the number of function
calls, and compiles faster because there are fewer template
instantiations.
…izeMany()

Instead of recursively calling `UnserializeMany` and peeling off one
argument at a time, use a fold expression. This simplifies the code,
makes it most likely faster because it reduces the number of function
calls, and compiles faster because there are fewer template
instantiations.
This gets rid of unnecessarily creating a temporary object T() to call
the right function.
This gets rid of unnecessarily creating a temporary object T() to call
the right function.
This gets rid of unnecessarily creating a temporary object T() to call
the right function.
This gets rid of unnecessarily creating a temporary object T() to call
the right function.
703b758 qa: Close SQLite connection properly (Hennadii Stepanov)

Pull request description:

  This PR is a follow-up for bitcoin/bitcoin#26462 that introduced a bug on Windows:
  ```
  >test\functional\wallet_descriptor.py
  ...
  PermissionError: [WinError 32] The process cannot access the file because it is being used by another process:
  ...
  ```

  From `sqlite3` Python module [docs](https://docs.python.org/3/library/sqlite3.html#how-to-use-the-connection-context-manager):
  > `Connection` object used as context manager only commits or rollbacks transactions, so the connection object should be closed manually.

ACKs for top commit:
  MarcoFalke:
    lgtm ACK 703b758
  theStack:
    utACK 703b758

Tree-SHA512: 35b1403507be06d1fc04e7e07ff56af5bcfe5013024671f0c1d9f3c41aacc4c777bcc6376ce82d720394e27450415d50ff5d5834ed388ec3f21503f86f1a42a5
…iables signed

92de74e refactor: Make more transaction size variables signed (Hennadii Stepanov)

Pull request description:

  This PR is a continuation of bitcoin/bitcoin#23962 and it:
  - gets rid of two static casts,
  - addresses bitcoin/bitcoin#23962 (comment),
  - is useful for bitcoin/bitcoin#25972, see the failed ARM and multiprocess CI jobs.

ACKs for top commit:
  MarcoFalke:
    lgtm ACK 92de74e  🥔
  glozow:
    ACK 92de74e

Tree-SHA512: 84225961af8e08439664e75661b98fe86560217e891e5633a28316bf248d88df317a0c6b5a5f6b03feb2b0e0fd40a1f91dd4a85a0610d567470805bf47a84487
Rather than using a bloom filter to track announced invs, simply allow
a peer to request any tx that entered the mempool prior to the last INV
message we sent them. This also obsoletes the UNCONDITIONAL_RELAY_DELAY.
We no longer have m_recently_announced_invs, so there is no need to add
txids to m_tx_inventory_known_filter to dedupe that filter.
Check that peers can immediately request txs from blocks that have been
reorged out and are now in our mempool.
Using `py.exe` launcher might by fragile depending on how Python was
installed. Specifying the Python version explicitly fixes test errors
like this:
```
RunCommandParseJSON error: process(py C:\Users\hebasto\bitcoin\test\functional\mocks\signer.py enumerate) returned 9009: Python was not found...
```

-BEGIN VERIFY SCRIPT-
sed -i 's|"py "|"py -3 "|g' $(git grep -l '"py "' -- test/functional)
-END VERIFY SCRIPT-
…ble` when possible

025fda0 fuzz: addrman, avoid `ConsumeDeserializable` when possible (brunoerg)

Pull request description:

  Using specific functions like `ConsumeService`, `ConsumeAddress` and `ConsumeNetAddr` may be more effective than using `ConsumeDeserializable`. They always return some value while `ConsumeDeserializable` may return `std::nullopt`.

  E.g.: In this part of the code, if `op_net_addr` is `std::nullopt`,  we basically generated the addresses (if so) unnecessarily, because we won't be able to use them:
  ```cpp
  std::vector<CAddress> addresses;
  LIMITED_WHILE(fuzzed_data_provider.ConsumeBool(), 10000) {
      const std::optional<CAddress> opt_address = ConsumeDeserializable<CAddress>(fuzzed_data_provider);
      if (!opt_address) {
          break;
      }
      addresses.push_back(*opt_address);
  }
  const std::optional<CNetAddr> opt_net_addr = ConsumeDeserializable<CNetAddr>(fuzzed_data_provider);
  if (opt_net_addr) {
      addr_man.Add(addresses, *opt_net_addr, std::chrono::seconds{ConsumeTime(fuzzed_data_provider, 0, 100000000)});
  }
  ```

  Also, if we are not calling `Add` effectively, it would also be affect other functions that may "depend" on it.

ACKs for top commit:
  dergoegge:
    Code review ACK 025fda0

Tree-SHA512: 02450bec0b084c15ba0cd1cbdfbac067c8fea4ccf27be0c86d54e020f029a6c749a16d8e0558f9d6d35a7ca9db8916f180c872f09474702b5591129e9be0d192
…carriersize=2 tests

faafc35 doc: Clarify that -datacarriersize applies to the full raw scriptPubKey, not the data push (MarcoFalke)
55550e7 test: Add -datacarriersize=2 tests (MarcoFalke)

Pull request description:

  Clarify with a test that `-datacarriersize` applies to the serialized size of the scriptPubKey, not the size of the pushed data. So for example,

  * `-datacarriersize=2` will reject a `raw(6a01aa)`, even though only one byte is pushed
  * `-datacarriersize=0` (or `-datacarrier=0`) will reject a `raw(6a)`, even though no byte is pushed
  * `-datacarriersize=0` (or `-datacarrier=0`) will reject a `raw(6a00)`, even though zero bytes are pushed

ACKs for top commit:
  ajtowns:
    ACK faafc35
  instagibbs:
    ACK bitcoin/bitcoin@faafc35

Tree-SHA512: f01ace02798f596ac2a02461e9f2a6ef91b3b37c976ea0b3bc860e2d3efb0ace0fd8b779dd18249cee7f84ebbe5fd21d8506afd3a15edadc00b843ff3b4aacc7
Enable `bitcoin-unterminated-logprintf`.

Co-authored-by: Cory Fields <cory-nospam-@coryfields.com>
fa47439 ci: Add missing linux-headers package to ASan task (MarcoFalke)
fabaa85 ci: Move ASan USDT to persistent_worker (MarcoFalke)

Pull request description:

  To run the USDT functional tests, the ASan task currently requires the container host to run the Ubuntu Lunar Linux kernel (or later). Cirrus CI is the only provider that allows to spin up full VMs with Ubuntu Lunar, however they will start to charge for all tasks (See slightly related discussion in bitcoin/bitcoin#28098).

  Since it is cheaper and recommended by Cirrus CI to just run a persistent worker, do that.

  Also, using a persistent worker allows to make use of the docker image cache.

ACKs for top commit:
  hebasto:
    ACK fa47439, I have reviewed the code and it looks OK.

Tree-SHA512: afd084ab1b56cbc3fa44d4611aaa01ec21c1d80aedf1f5f1bc4b8b3d1bd08095e0c7fcea7a3e6ec4b6cd97d01e97ee86061eb84a5e2c7e7195ce02a186254900
Connman uses this new map to keep a count of active OUTBOUND_FULL_RELAY and
MANUAL connections. Unused until next commit.

Co-authored-by: Martin Zumsande <mzumsande@gmail.com>
If a peer is the only one of its network, protect it from eviction.
This improves the diversity of outbound connections with respect to
reachable networks.

Co-authored-by: Amiti Uttarwar <amiti@uttarwar.org>
Co-authored-by: Amiti Uttarwar <amiti@uttarwar.org>
Diversify outbound connections with respect to
networks: Every ~5 minutes, try to add an extra connection
to a reachable network which we currently don't have a connection to.
This is done defensively - only try management with respect to networks
after all existing outbound slots are filled.
The resulting situation with an extra outbound peer will be handled
by the extra outbound eviction logic, which protects peers from
eviction if they are the only ones for their network.

Co-authored-by: Amiti Uttarwar <amiti@uttarwar.org>
It is incorrect to assert that `cache.HaveCoin()` will always be `true`
if `backend.HaveCoin()` is. The coin could well have been marked as
spent in the cache but not yet flushed, in which case `cache.HaveCoin()`
would return `false`.

Note this was never hit because `exists_using_have_coin_in_backend` is
currently never `true` (it's the default implementation of `CCoinsView`.
However this might change if we were to add a target where the backend
is a `CCoinsViewDB`.
f054bd0 refactor: use "if constexpr" in std::vector's Unserialize() (Martin Leitner-Ankerl)
088caa6 refactor: use "if constexpr" in std::vector's Serialize() (Martin Leitner-Ankerl)
0fafaca refactor: use "if constexpr" in prevector's Unserialize() (Martin Leitner-Ankerl)
c8839ec refactor: use "if constexpr" in prevector's Serialize() (Martin Leitner-Ankerl)
1403d18 refactor: use fold expressions instead of recursive calls in UnserializeMany() (Martin Leitner-Ankerl)
bd08a00 refactor: use fold expressions instead of recursive calls in SerializeMany() (Martin Leitner-Ankerl)

Pull request description:

  This simplifies the serialization code a bit and should also make it a bit faster.

  * use fold expressions instead of recursive calls. This simplifies the code, makes it most likely faster because it reduces the number of function calls, and compiles faster because there are fewer template instantiations.

  * use `if constexpr` instead of unnecessarily creating a temporary object only to call the right overload. This is used for `std::vector` and `prevector` serialization.

ACKs for top commit:
  MarcoFalke:
    only change is to add a missing `&`. lgtm, re-ACK f054bd0 📦
  jonatack:
    ACK f054bd0
  sipa:
    utACK f054bd0
  john-moffett:
    ACK f054bd0

Tree-SHA512: 0417bf2d6be486c581732297945449211fc3481bac82964e27628b38ef55a47dfa58d730148aeaf1b19fa8eb1076489cc646ceebb178162a9afa59034601501d
fanquake and others added 18 commits August 29, 2023 09:27
…hains

13eb8aa doc: Release notes for testnet defaulting to -acceptnonstdtxn=0 (Anthony Towns)
e1dc15d config: default acceptnonstdtxn=0 on all chains (Anthony Towns)

Pull request description:

  Changes `-acceptnonstxtxn` to default to 0 on testnet, matching the other chains. Allowing non-standard txs on testnet by default contributed to the difficulties RSK described in #26348: "We see that there are two script paths and, to reduce the script size, a single CHECKMULTISIG is used for the two paths, separating the signer count from the CHECKMULTISIG opcode. This script worked on testnet, because it lacks the standard checks performed in Mainnet."

ACKs for top commit:
  MarcoFalke:
    lgtm ACK 13eb8aa
  sipa:
    utACK 13eb8aa
  instagibbs:
    utACK bitcoin/bitcoin@13eb8aa
  theStack:
    Code-review ACK 13eb8aa

Tree-SHA512: eff7a3f9fc9b94003a730beb96e6f3399bc8b8e93fde4b15f20a11eda61d9a3e076f4423989f98b794b32681abecbc3756a54cd0d37b136e2fb2ffbb47ee7774
…ith importlib.metadata

6c008a2 script: replace deprecated pkg_resources with importlib.metadata (Jon Atack)

Pull request description:

  Running our python linter with a recent python and the latest release of setuptools [v68.1.2](https://setuptools.pypa.io/en/stable/history.html):

  ```
  $ python3 --version
  Python 3.11.5
  $ ./test/lint/lint-python.py:12: DeprecationWarning: pkg_resources is deprecated as an API.
    See https://setuptools.pypa.io/en/latest/pkg_resources.html
    import pkg_resources
  ```

  Using `pkg_resources` was [deprecated](pypa/setuptools#3843) earlier in [v67.5.0](https://setuptools.pypa.io/en/stable/history.html#id55): "Although pkg_resources has been discouraged for use, some projects still consider pkg_resources viable for usage. This change makes it clear that pkg_resources should not be used, emitting a DeprecationWarning when imported."

  The `importlib.metadata` library requires Python 3.8, which is currently our minimum-supported Python version.

  For more details about `importlib.metadata` and the two methods imported and used here, see:

  - https://docs.python.org/3/library/importlib.metadata.html
  - https://importlib-metadata.readthedocs.io/en/latest/api.html#importlib_metadata.metadata
  - https://importlib-metadata.readthedocs.io/en/latest/api.html#importlib_metadata.PackageNotFoundError

ACKs for top commit:
  MarcoFalke:
    lgtm ACK 6c008a2 (review only, did not test)

Tree-SHA512: f5258d37043fcc9744f85641a60a3395ad43822c72d030dea8c39fa7f48ec3d7790cdeeb832f96e8f38046adb7c62fbc577c975ef0c77c8047c0c8f2353ce540
@hebasto
Copy link
Member Author

hebasto commented Aug 29, 2023

@MarcoFalke

After the last push, the CI merge_base_script works in a wrong way:

bash -c "$PACKAGE_MANAGER_INSTALL git"
Hit:1 https://mirror.hetzner.com/ubuntu/packages lunar InRelease
Hit:2 https://mirror.hetzner.com/ubuntu/packages lunar-updates InRelease
Hit:3 https://mirror.hetzner.com/ubuntu/packages lunar-backports InRelease
Hit:4 https://mirror.hetzner.com/ubuntu/security lunar-security InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
git is already the newest version (1:2.39.2-1ubuntu1.1).
0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
if [ "$CIRRUS_PR" = "" ]; then exit 0; fi
git fetch --depth=1 $CIRRUS_REPO_CLONE_URL "pull/${CIRRUS_PR}/merge"
From https://github.com/bitcoin-core/gui-qml
* branch            refs/pull/359/merge -> FETCH_HEAD
git checkout FETCH_HEAD
Previous HEAD position was 592da16 Adjust code according to changes in the main repository
HEAD is now at 6f6e8b4 Merge b1f441ff3c280b4a59ccd2c15658628877830068 into 67895f77f2da6471c5c3b67988fde7f932f77782
M	ci/test/00_setup_env_native_asan.sh

The HEAD must be at Merge 592da1615079d98afd81c074f0447122b718b125 into 67895f77f2da6471c5c3b67988fde7f932f77782, no?

UPD. Or is it just an intermittent GitHub issue?

@hebasto
Copy link
Member Author

hebasto commented Aug 29, 2023

UPD. Or is it just an intermittent GitHub issue?

Looks like that. Re-run fixes it.

UPD. It still happens.

@hebasto
Copy link
Member Author

hebasto commented Aug 31, 2023

Going to merge this PR to unlock migration from non-free Cirrus CI to free GitHub Actions.

Hoping someone will make post-merge review soon.

@hebasto hebasto merged commit 14ddf61 into bitcoin-core:main Aug 31, 2023
12 checks passed
@hebasto hebasto deleted the 230829-sync branch August 31, 2023 22:07
@promag
Copy link
Contributor

promag commented Aug 31, 2023

utACK 14ddf61.

ACK merge commit b7cdd31

Light code review:

  • b326114 ci: Disable the ARM task temporarily
  • 2719ce1 Fix modernize-use-default-member-init clang-tidy warnings
  • 8b02563 Adjust code according to changes in the main repository
  • 81643a5 ci, qml: Add required Qt Quick dev packages
  • dea9e51 build, qt: Enable OpenGL for Windows static builds
  • 3d78b7a qml, build: Update PE_ALLOWED_LIBRARIES for Qt 5.15.2
  • 16f77a5 net: Notify about connection number by type

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet