Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Schnorr Checksig / CheckDatasig Modifications to the Scripting System #134

Open
wants to merge 23 commits into
base: master
Choose a base branch
from

Conversation

jonajosejg
Copy link
Contributor

@jonajosejg jonajosejg commented Apr 23, 2019

Implemented Schnorr Signature Checks on the following Opcodes:
OP_CHECKSIG / OP_CHECKDATSIG.

Along with MTP Activation for the Hardfork found in the chain.js file (perhaps these commits shouldn't have been added in conjunction with these particular commits).

In which case, I'd be more then willing to close this and re-commit the PR with the Segwit Recovery changes in a separate branch as well.

I will be adding extra primitives to the script-test.js file found in the Unit-Test directory. Along with Segwit Recovery checks to the verify functions in the script.js file.

@jonajosejg
Copy link
Contributor Author

The reason for CI failures are due to the remaining Segwit Recovery test failures I will close this and recommit it. So it's more organized @tuxcanfly could I ask for pointers involving the segwit recovery function found at:

https://github.com/Bitcoin-ABC/bitcoin-abc/blob/193d2b9c296a05f64db546093df127f463b36490/src/script/interpreter.cpp#L1645

@pinheadmz
Copy link
Member

@alwaysAn0n check this out

@@ -3112,6 +3134,11 @@ class Script {
const raw = stack.pop();
const redeem = Script.fromRaw(raw);

if (!(flags & Script.flags.VERIFY_SEGWIT_RECOVERY) !== 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you need to check for the flag to be set, the ! is doing the opposite. Also the reason for some failing tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're the man , thanks getting rid of that symbol corrects all the failing tests for Segwit Recovery.

@jonajosejg
Copy link
Contributor Author

jonajosejg commented Apr 25, 2019

Hey guy's im not trying to make things confusing here, but if you fork the software with my schnorr-branch and run all the script-tests with bmocha they all pass. Im not sure as to why ci is throwing assert.ifError on 30 tests with Schnorr.

@pinheadmz
Copy link
Member

@tuxcanfly or @nodar-chkuaselidze can you guys check the Travis CI script is correct?
I am experiencing same output as Rojii -- tests pass locally:
npm run test
npm run test-ci

@@ -1131,7 +1131,7 @@ class Script {
|| !(sig[sig.length - 1] & Script.hashType.SIGHASH_FORKID))
subscript.findAndDelete(sig);

validateTXSignature(sig, flags);
CheckTransactionSignature(sig, flags);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Function names are generally camel cased with the first letter being lowercase in this codebase. There are multiple functions that are camel cased that start with uppercased letters

@tynes
Copy link
Member

tynes commented Apr 25, 2019

Please be sure to fix the linting errors and its important to make the tests pass in CI. Have you tried syncing the node against the upgraded testnet?

@jonajosejg
Copy link
Contributor Author

I’ll fix it now

@tuxcanfly
Copy link
Member

@tuxcanfly or @nodar-chkuaselidze can you guys check the Travis CI script is correct?
I am experiencing same output as Rojii -- tests pass locally:
npm run test
npm run test-ci

looks like its failing on node 10.2.0. If you don't want to fix that, I believe you will need to make the Circle CI workspace upgrade to node 11. @nodar-chkuaselidze can help with that.

@tuxcanfly
Copy link
Member

Please be sure to fix the linting errors and its important to make the tests pass in CI. Have you tried syncing the node against the upgraded testnet?

Looks like there are some remaining lint issues:

/home/tuxcanfly/Work/bcash/lib/script/script.js
  3090:10  error  Expected to return a value at the end of static method 'verify'  consistent-return
  3299:42  error  Missing space before opening brace                               space-before-blocks
  3413:50  error  Missing semicolon                                                semi

@pinheadmz
Copy link
Member

@tuxcanfly Tests passed locally for me after switching to node 10.2, I saw that version in the CI log

@tuxcanfly
Copy link
Member

@tuxcanfly Tests passed locally for me after switching to node 10.2, I saw that version in the CI log

As it happens, when I switched to 10.2 I still had packages installed from 11.0. Looking at the CI logs:

No cache is found for key: v1-dependencies-GFWM2v9feEhJXXOj1B7pCmUoK1zt5CDI22HU6SXcalU=
Found a cache from build 571 at v1-dependencies-

So it looks like the CI is fetching the dependencies from a previous build which probably has no schnorr package built.

As to why it's not saving the cache and restoring it, I have no clue 🤷‍♂️

@tuxcanfly
Copy link
Member

Hmm, it could be that npm install istanbul codecov etc step is changing the package.json file, so it's checksum fails to match during the save cache phase?

https://support.circleci.com/hc/en-us/articles/360004632473-No-Cache-Found-and-Skipping-Cache-Generation

If this file is modified in any way between the restore and save steps, you will have mismatching keys, meaning you are attempting to save your cache to one key, and restore from another, which will always fail.

@nodech
Copy link
Member

nodech commented Apr 28, 2019

It seems that current version of the bcrypto is not building secp256k1 correctly on circle ci version.
gmp.h not found.

../src/secp256k1/src/num_gmp.h:10:17: fatal error: gmp.h: No such file or directory
 #include <gmp.h>

--

This causes bcrypto to fall back to js.
So the problem is in the bcrypto js version.

You can reproduce: NODE_BACKEND=js npm run test.

--
It is recovering cache because it uses last package.json, so node_modules is the same so no reason to reinstall. Even if you cleaned the cache, result would be the same.

--
Notes on build failure:
Build failure started from bcrypto@3.1.3, if you downgrade to 3.1.2, it will at least run tests against native, even though JS version will still fail.

-- UPDATE

Update: both problems should have been fixed in bcrypto.
So next release should solve the problem.

@nodech
Copy link
Member

nodech commented Apr 28, 2019

The reason why build fails after 3.1.2: bcoin-org/bcrypto#16

@codecov-io
Copy link

codecov-io commented May 7, 2019

Codecov Report

Merging #134 into master will decrease coverage by 0.04%.
The diff coverage is 97.33%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #134      +/-   ##
==========================================
- Coverage   55.55%   55.51%   -0.05%     
==========================================
  Files         109      109              
  Lines       27184    27238      +54     
  Branches     4498     4513      +15     
==========================================
+ Hits        15102    15121      +19     
- Misses      12082    12117      +35
Impacted Files Coverage Δ
lib/net/seeds/main.js 100% <ø> (ø) ⬆️
lib/net/seeds/testnet.js 100% <ø> (ø) ⬆️
lib/protocol/networks.js 100% <ø> (ø) ⬆️
lib/net/common.js 85.18% <100%> (ø) ⬆️
lib/script/script.js 78.76% <100%> (+0.67%) ⬆️
lib/blockchain/chain.js 62.81% <100%> (+0.29%) ⬆️
lib/mempool/mempool.js 54.57% <66.66%> (-0.01%) ⬇️
lib/script/common.js 80% <66.66%> (-1.06%) ⬇️
lib/primitives/invitem.js 27.27% <0%> (-9.1%) ⬇️
lib/utils/binary.js 53.84% <0%> (-5.13%) ⬇️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update faf1eeb...383a93d. Read the comment docs.

@andrewcottage
Copy link

How's this going? Anything I can do to help?

@tynes
Copy link
Member

tynes commented May 13, 2019

@andrewcottage Thanks for being interested in helping out Andrew! More review can always help and also verifying that it can sync on the Schnorr Testnet. Also verifying that all of the tests from Bitcoin ABC were ported over properly

@jonajosejg
Copy link
Contributor Author

Utilize this branch if you're going to sync the lastest testnet
https://github.com/rojii/bcash/commits/testnet-branch

in the readme explains what to do, there's a bcash.conf file that you would like to add in your .bcash directory where the chain is going to stored locally.

@jonajosejg
Copy link
Contributor Author

This is going to be my last day in adding the rest of the Stack Tests, for regular schnorr transactions / segwit recovery.

Feel free to ask any questions

@andrewcottage
Copy link

Sounds good. I'm having an internet outage, so I won't be able to sync the chain tonight (tethering to my phone currently). I might be able to do that tomorrow along with comparing tests.

@andrewcottage
Copy link

I keep getting


[91515:0x102803600]  2597451 ms: Mark-sweep 1397.8 (1435.8) -> 1397.7 (1436.3) MB, 870.7 / 7.5 ms  allocation failure GC in old space requested
[91515:0x102803600]  2598462 ms: Mark-sweep 1397.7 (1436.3) -> 1397.7 (1429.3) MB, 1011.7 / 8.5 ms  last resort GC in old space requested
[91515:0x102803600]  2599490 ms: Mark-sweep 1397.7 (1429.3) -> 1397.7 (1429.3) MB, 1027.3 / 7.6 ms  last resort GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2e48c67257c1 <JSObject>
    1: set(this=0x2e4811e24df1 <Map map = 0x2e48036848d9>,0x2e4867732a61 <String[32]\: (\xde}\xbe\xcd\x83\xefk\xc4\xca\x92\xfb\xd7@\xd4\x92\xad\x92\xe5\x1b\xb7\xae\xed\xe0J#\x00\x00\x00\x00\x00\x00>,0x2e4867732a99 <BufferItem map = 0x2e48127d8789>)
    2: getBlock [/Users/andrewcottage/Projects/open_source/bcash/lib/net/pool.js:~3202] [pc=0x13a4264592c2](this=0x2e483e267da9 <EventEmitter map = 0...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [/Users/andrewcottage/.nvm/versions/node/v8.11.0/bin/node]
 2: node::FatalException(v8::Isolate*, v8::Local<v8::Value>, v8::Local<v8::Message>) [/Users/andrewcottage/.nvm/versions/node/v8.11.0/bin/node]
 3: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/Users/andrewcottage/.nvm/versions/node/v8.11.0/bin/node]
 4: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [/Users/andrewcottage/.nvm/versions/node/v8.11.0/bin/node]
 5: v8::internal::OrderedHashTable<v8::internal::OrderedHashMap, 2>::Rehash(v8::internal::Handle<v8::internal::OrderedHashMap>, int) [/Users/andrewcottage/.nvm/versions/node/v8.11.0/bin/node]
 6: v8::internal::Runtime_MapGrow(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/andrewcottage/.nvm/versions/node/v8.11.0/bin/node]
 7: 0x13a4263042fd
Abort trap: 6

When trying to sync the chain. I tried both main net and testnet

@jonajosejg
Copy link
Contributor Author

This is due to running the tx / addr indexers I’ll be porting over the changes , bug fixes made to bcoin in my branch in the next day


if (await tx.verifyAsync(view, flags, this.workers)) {
throw new VerifyError(tx,
'nonstandard',
'non-mandatory-script-verify-flag',
0);
}

if (await tx.verifyAsync(view,

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In principle this stuff can be take out now that the upgrade has happened.

Copy link

@markblundeberg markblundeberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see some unrelated stuff like changes to MAX_MESSAGE, seeds list, checkpoints ... can these not be in separate PRs given that the topic of this PR is Schnorr sigs?

@markblundeberg
Copy link

markblundeberg commented May 20, 2019

Overall it looks like the code is doing the right things related to the new activated features, however I don't know javascript well so it's hard to say clearly.

The fact that this synced to the fork testnet and now the main chain is a actually good sign, given that the new features are quite minimal changes. I have noticed on discussions that there were issues with bcash, but they appear to be unrelated to these changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants