Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add compile time verification of assumptions we're currently making implicitly/tacitly #15391

Merged
merged 1 commit into from Feb 15, 2019

Conversation

@practicalswift
Copy link
Contributor

@practicalswift practicalswift commented Feb 12, 2019

Add compile time verification of assumptions we're currently making implicitly/tacitly.

As suggested by @sipa in #14239 (comment) and @MarcoFalke in #14479 (comment).

@gmaxwell
Copy link
Contributor

@gmaxwell gmaxwell commented Feb 12, 2019

It would be useful for it to get compiled, at least AFAICT adding a false assumption here won't make it fail. :) Concept ACK. Maybe also the #if defined(NDEBUG)? check? Probably every other primitive type we depend on the size of, including the unsigned ones.

@MarcoFalke
Copy link
Member

@MarcoFalke MarcoFalke commented Feb 12, 2019

You are only adding a header. Does this need to be included in a cpp file to get compiled?

@practicalswift
Copy link
Contributor Author

@practicalswift practicalswift commented Feb 12, 2019

@gmaxwell @MarcoFalke Yes, obviously it needs to be included :-) The inclusion somehow got lost during my latest git commit --amend fixup. Fixing!

@practicalswift
Copy link
Contributor Author

@practicalswift practicalswift commented Feb 12, 2019

Now including from src/util/system.h which also is the most included file FWIW :-)

$ git grep -E "^#include " -- "*.cpp" | cut -f2 -d'<' | cut -f1 -d'>' | sort | uniq -c | \
      sort -n | tail -1
     99 util/system.h

Let me know if you can think of a more appropriate file for the include.

@practicalswift practicalswift force-pushed the assumptions branch 3 times, most recently from a015e45 to 8add86e Feb 13, 2019
@practicalswift
Copy link
Contributor Author

@practicalswift practicalswift commented Feb 13, 2019

Added a couple of assumptions and listed important "non-assumptions".

Please help me identify further assumptions and corresponding examples of where we are relying on said assumptions :-)

@jb55
Copy link
Contributor

@jb55 jb55 commented Feb 14, 2019

utACK 8add86e

@practicalswift
Copy link
Contributor Author

@practicalswift practicalswift commented Feb 14, 2019

@jb55 Thanks for the review! Can you think of any further assumptions and examples of where we rely on them being true? :-)

src/assumptions.h Outdated Show resolved Hide resolved
@practicalswift
Copy link
Contributor Author

@practicalswift practicalswift commented Feb 14, 2019

Moved to src/compat/ and added an explicit non-assumption regarding size_t :-)

@jb55
Copy link
Contributor

@jb55 jb55 commented Feb 14, 2019

re-utACK 7548e6e

@laanwj
Copy link
Member

@laanwj laanwj commented Feb 14, 2019

utACK 7548e6e
I'm not 100% sure we make the int=32 bit assumption (more like "int is at least 32 bit" I think? otherwise we use explicitly sized types like int32_t), but I doubt anyone ever tested the code on an architecture with a different integer size so I'm fine with making the assumption.

@practicalswift
Copy link
Contributor Author

@practicalswift practicalswift commented Feb 14, 2019

@laanwj If I'm reading GetSizeOfCompactSize, WriteCompactSize and ReadCompactSize correctly we're assuming that int has a width of exactly 32 bits, no?

Example:

/**
 * Compact Size
 * size <  253        -- 1 byte
 * size <= USHRT_MAX  -- 3 bytes  (253 + 2 bytes)
 * size <= UINT_MAX   -- 5 bytes  (254 + 4 bytes)
 * size >  UINT_MAX   -- 9 bytes  (255 + 8 bytes)
 */
inline unsigned int GetSizeOfCompactSize(uint64_t nSize)
{
    if (nSize < 253)             return sizeof(unsigned char);
    else if (nSize <= std::numeric_limits<unsigned short>::max()) return sizeof(unsigned char) + sizeof(unsigned short);
    else if (nSize <= std::numeric_limits<unsigned int>::max())  return sizeof(unsigned char) + sizeof(unsigned int);
    else                         return sizeof(unsigned char) + sizeof(uint64_t);
}

@practicalswift
Copy link
Contributor Author

@practicalswift practicalswift commented Feb 14, 2019

@jb55 @laanwj Please re-review after s/BITCOIN_ASSUMPTIONS_H/BITCOIN_COMPAT_ASSUMPTIONS_H/g

@laanwj
Copy link
Member

@laanwj laanwj commented Feb 14, 2019

@laanwj If I'm reading GetSizeOfCompactSize, WriteCompactSize and ReadCompactSize correctly we're assuming that int has a width of exactly 32 bits?

You're right, thanks for giving an example.

@MarcoFalke
Copy link
Member

@MarcoFalke MarcoFalke commented Feb 14, 2019

Would it make sense to refer to an example for each assumption. That way, we know of at least one example. An alternative would be to just inline the assumptions where they are needed.

@practicalswift
Copy link
Contributor Author

@practicalswift practicalswift commented Feb 14, 2019

@MarcoFalke I'm not sure I follow: the examples have been there since this PR first was submitted? :-)

In this specific case the following has been in there all along:

// Assumption: We assume integer widths.
// Example(s): GetSizeOfCompactSize and WriteCompactSize in the serialization
//             code.
static_assert(sizeof(short) == 2, "16-bit short assumed");
static_assert(sizeof(int) == 4, "32-bit int assumed");

:-)

@MarcoFalke
Copy link
Member

@MarcoFalke MarcoFalke commented Feb 14, 2019

Ok, my bad. I must have missed them when I last looked at it a few days ago.

@MarcoFalke
Copy link
Member

@MarcoFalke MarcoFalke commented Feb 14, 2019

ACK 7cee858

@DrahtBot
Copy link
Contributor

@DrahtBot DrahtBot commented Feb 14, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #15146 (Solve SmartOS FD_ZERO build issue by Empact)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@sipa
Copy link
Member

@sipa sipa commented Feb 14, 2019

utACK 7cee858

@laanwj laanwj merged commit 7cee858 into bitcoin:master Feb 15, 2019
2 checks passed
laanwj added a commit that referenced this issue Feb 15, 2019
…ently making implicitly/tacitly

7cee858 Add compile time verification of assumptions we're currently making implicitly/tacitly (practicalswift)

Pull request description:

  Add compile time verification of assumptions we're currently making implicitly/tacitly.

  As suggested by @sipa in #14239 (comment) and @MarcoFalke in #14479 (comment).

Tree-SHA512: e68fe51164dbd3eeb76aa8a7e83dfcd3b4d5a66037c0f1822bbbd189bbe3c280e03b3b10af870880ecc09b612e62fb3d9bcd6cf1e16cb7ba818c257db0712ce4
MarcoFalke added a commit that referenced this issue Feb 17, 2019
…t file

3ec56be appveyor: Remove unused NDEBUG removal (Chun Kuan Lee)
8a1f0a3 scripted-diff: Remove NDEBUG pre-define (Chun Kuan Lee)

Pull request description:

  Follow #15391

Tree-SHA512: f264418cbc69b5f083469ed9005a6d592d4268f2b7da967e571ce30195de73b09a9e14c8610a5b6b0f056847d82a4bc7c2fbe56498307093aab4dd42903e6137
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Sep 27, 2019
…mplicitly/tacitly

Summary:
Add compile time verification of assumptions we're currently making implicitly/tacitly.

Backport of Bitcoin Core PR15391
bitcoin/bitcoin#15391

Test Plan:
```
make check
```

Reviewers: Fabien, #bitcoin_abc, deadalnix

Reviewed By: Fabien, #bitcoin_abc

Differential Revision: https://reviews.bitcoinabc.org/D4043
jonspock added a commit to jonspock/devault that referenced this issue Dec 24, 2019
…mplicitly/tacitly

Summary:
Add compile time verification of assumptions we're currently making implicitly/tacitly.

Backport of Bitcoin Core PR15391
bitcoin/bitcoin#15391

Test Plan:
```
make check
```

Reviewers: Fabien, #bitcoin_abc, deadalnix

Reviewed By: Fabien, #bitcoin_abc

Differential Revision: https://reviews.bitcoinabc.org/D4043
jonspock added a commit to jonspock/devault that referenced this issue Dec 24, 2019
…mplicitly/tacitly

Summary:
Add compile time verification of assumptions we're currently making implicitly/tacitly.

Backport of Bitcoin Core PR15391
bitcoin/bitcoin#15391

Test Plan:
```
make check
```

Reviewers: Fabien, #bitcoin_abc, deadalnix

Reviewed By: Fabien, #bitcoin_abc

Differential Revision: https://reviews.bitcoinabc.org/D4043
jonspock added a commit to jonspock/devault that referenced this issue Dec 24, 2019
…mplicitly/tacitly

Summary:
Add compile time verification of assumptions we're currently making implicitly/tacitly.

Backport of Bitcoin Core PR15391
bitcoin/bitcoin#15391

Test Plan:
```
make check
```

Reviewers: Fabien, #bitcoin_abc, deadalnix

Reviewed By: Fabien, #bitcoin_abc

Differential Revision: https://reviews.bitcoinabc.org/D4043
jonspock added a commit to jonspock/devault that referenced this issue Dec 24, 2019
…mplicitly/tacitly

Summary:
Add compile time verification of assumptions we're currently making implicitly/tacitly.

Backport of Bitcoin Core PR15391
bitcoin/bitcoin#15391

Test Plan:
```
make check
```

Reviewers: Fabien, #bitcoin_abc, deadalnix

Reviewed By: Fabien, #bitcoin_abc

Differential Revision: https://reviews.bitcoinabc.org/D4043
jonspock added a commit to jonspock/devault that referenced this issue Dec 24, 2019
…mplicitly/tacitly

Summary:
Add compile time verification of assumptions we're currently making implicitly/tacitly.

Backport of Bitcoin Core PR15391
bitcoin/bitcoin#15391

Test Plan:
```
make check
```

Reviewers: Fabien, #bitcoin_abc, deadalnix

Reviewed By: Fabien, #bitcoin_abc

Differential Revision: https://reviews.bitcoinabc.org/D4043
jonspock added a commit to jonspock/devault that referenced this issue Dec 24, 2019
…mplicitly/tacitly

Summary:
Add compile time verification of assumptions we're currently making implicitly/tacitly.

Backport of Bitcoin Core PR15391
bitcoin/bitcoin#15391

Test Plan:
```
make check
```

Reviewers: Fabien, #bitcoin_abc, deadalnix

Reviewed By: Fabien, #bitcoin_abc

Differential Revision: https://reviews.bitcoinabc.org/D4043
jonspock added a commit to devaultcrypto/devault that referenced this issue Dec 26, 2019
…mplicitly/tacitly

Summary:
Add compile time verification of assumptions we're currently making implicitly/tacitly.

Backport of Bitcoin Core PR15391
bitcoin/bitcoin#15391

Test Plan:
```
make check
```

Reviewers: Fabien, #bitcoin_abc, deadalnix

Reviewed By: Fabien, #bitcoin_abc

Differential Revision: https://reviews.bitcoinabc.org/D4043
MarcoFalke added a commit that referenced this issue Jul 2, 2020
…se of uninitialized memory

870f0cd build: Add MemorySanitizer (MSan) in Travis to detect use of uninitialized memory (practicalswift)

Pull request description:

  Add MemorySanitizer (MSan) in Travis to detect use of uninitialized memory.

  First UBSan, then ASan followed by TSan... and now: yes, the wait is over -- **MSan is finally here!** :)

  Some historical context:
  * 2017: Continuous compilation with Clang Thread Safety analysis enabled (#10866, #10923)
  * 2018: Continuous testing with trapping on signed integer overflows (`-ftrapv`) (#12686)
  * 2018: Continuous testing of use of locale dependent functions (#13041)
  * 2018: Continuous testing of format strings (#13705)
  * 2018: Continuous compilation with MSVC `TreatWarningAsError` (#14151)
  * 2018: Continuous testing under UndefinedBehaviorSanitizer – UBSan (#14252, #14673, #17006)
  * 2018: Continuous testing under AddressSanitizer – ASan (#14794, #17205, #17674)
  * 2018: Continuous testing under ThreadSanitizer – TSan (#14829)
  * 2019: Continuous testing in an unsigned char environment (`-funsigned-char`) (#15134)
  * 2019: Continuous compile-time testing of assumptions we're making (#15391)
  * 2019: Continuous testing of fuzz test cases under Valgrind (#17633, #18159, #18166)
  * 2020: Finally... MemorySanitizer – MSAN! :)

  What is the next step? What tools should we add to CI to keep bugs from entering `master`? :)

ACKs for top commit:
  MarcoFalke:
    ACK 870f0cd

Tree-SHA512: 38327c8b75679d97d469fe42e704cacd1217447a5a603701dd8a58ee50b3be2c10248f8d68a479ed081c0c4b254589d3081c9183f991640b06ef689061f75578
@practicalswift practicalswift deleted the assumptions branch Apr 10, 2021
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jun 27, 2021
…re currently making implicitly/tacitly

7cee858 Add compile time verification of assumptions we're currently making implicitly/tacitly (practicalswift)

Pull request description:

  Add compile time verification of assumptions we're currently making implicitly/tacitly.

  As suggested by @sipa in bitcoin#14239 (comment) and @MarcoFalke in bitcoin#14479 (comment).

Tree-SHA512: e68fe51164dbd3eeb76aa8a7e83dfcd3b4d5a66037c0f1822bbbd189bbe3c280e03b3b10af870880ecc09b612e62fb3d9bcd6cf1e16cb7ba818c257db0712ce4
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jun 28, 2021
…re currently making implicitly/tacitly

7cee858 Add compile time verification of assumptions we're currently making implicitly/tacitly (practicalswift)

Pull request description:

  Add compile time verification of assumptions we're currently making implicitly/tacitly.

  As suggested by @sipa in bitcoin#14239 (comment) and @MarcoFalke in bitcoin#14479 (comment).

Tree-SHA512: e68fe51164dbd3eeb76aa8a7e83dfcd3b4d5a66037c0f1822bbbd189bbe3c280e03b3b10af870880ecc09b612e62fb3d9bcd6cf1e16cb7ba818c257db0712ce4
PastaPastaPasta added a commit to PastaPastaPasta/dash that referenced this issue Jun 29, 2021
…re currently making implicitly/tacitly

7cee858 Add compile time verification of assumptions we're currently making implicitly/tacitly (practicalswift)

Pull request description:

  Add compile time verification of assumptions we're currently making implicitly/tacitly.

  As suggested by @sipa in bitcoin#14239 (comment) and @MarcoFalke in bitcoin#14479 (comment).

Tree-SHA512: e68fe51164dbd3eeb76aa8a7e83dfcd3b4d5a66037c0f1822bbbd189bbe3c280e03b3b10af870880ecc09b612e62fb3d9bcd6cf1e16cb7ba818c257db0712ce4
vijaydasmp added a commit to vijaydasmp/dash that referenced this issue Oct 4, 2021
…etect use of uninitialized memory

870f0cd build: Add MemorySanitizer (MSan) in Travis to detect use of uninitialized memory (practicalswift)

Pull request description:

  Add MemorySanitizer (MSan) in Travis to detect use of uninitialized memory.

  First UBSan, then ASan followed by TSan... and now: yes, the wait is over -- **MSan is finally here!** :)

  Some historical context:
  * 2017: Continuous compilation with Clang Thread Safety analysis enabled (bitcoin#10866, bitcoin#10923)
  * 2018: Continuous testing with trapping on signed integer overflows (`-ftrapv`) (bitcoin#12686)
  * 2018: Continuous testing of use of locale dependent functions (bitcoin#13041)
  * 2018: Continuous testing of format strings (bitcoin#13705)
  * 2018: Continuous compilation with MSVC `TreatWarningAsError` (bitcoin#14151)
  * 2018: Continuous testing under UndefinedBehaviorSanitizer – UBSan (bitcoin#14252, bitcoin#14673, bitcoin#17006)
  * 2018: Continuous testing under AddressSanitizer – ASan (bitcoin#14794, bitcoin#17205, bitcoin#17674)
  * 2018: Continuous testing under ThreadSanitizer – TSan (bitcoin#14829)
  * 2019: Continuous testing in an unsigned char environment (`-funsigned-char`) (bitcoin#15134)
  * 2019: Continuous compile-time testing of assumptions we're making (bitcoin#15391)
  * 2019: Continuous testing of fuzz test cases under Valgrind (bitcoin#17633, bitcoin#18159, bitcoin#18166)
  * 2020: Finally... MemorySanitizer – MSAN! :)

  What is the next step? What tools should we add to CI to keep bugs from entering `master`? :)

ACKs for top commit:
  MarcoFalke:
    ACK 870f0cd

Tree-SHA512: 38327c8b75679d97d469fe42e704cacd1217447a5a603701dd8a58ee50b3be2c10248f8d68a479ed081c0c4b254589d3081c9183f991640b06ef689061f75578
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

7 participants