Merge pull request #214 from blockchain-certificates/v3

V3

.gitignore

@@ -20,6 +20,7 @@ data/work/*
 data*/*
 dist/*
 build/*
+node_modules/*
 
 requirements.txt
 .idea/*

README.md

@@ -5,9 +5,12 @@
 # cert-issuer
 
 The cert-issuer project issues blockchain certificates by creating a transaction from the issuing institution to the
-recipient on the Bitcoin blockchain that includes the hash of the certificate itself.
+recipient on the Bitcoin or Ethereum blockchains. That transaction includes the hash of the certificate itself.
 
-Note: Work on Blockcerts v3 is underway.  It will handle Blockcerts using the Verifiable Credentials standard, and sign them with MerkleProof2019.  Thus, cert-issuer v3 will _not_ be backward compatible - v2 Blockcerts will not issue from v3 cert-issuer.  
+Blockcerts v3 is released. This new version of the standard leverages the [W3C Verifiable Credentials specification](https://www.w3.org/TR/vc-data-model/), and documents are signed with [MerkleProof2019 LD signature](https://w3c-ccg.github.io/lds-merkle-proof-2019/). Use of [DIDs (Decentralized Identifiers)](https://www.w3.org/TR/did-core/) is also possible to provide more cryptographic proof of the ownership of the issuing address. See [section](#working-with-dids) down below
+
+Cert-issuer v3 is _not_ backwards compatible and does not support Blockcerts v2 issuances. If you need to work with v2, you need to install cert-issuer v2 or use the [v2](https://github.com/blockchain-certificates/cert-issuer/tree/v2) branch of this repo.
+You may expect little to no maintenance to the v2 code at this point.
 
 ## Web resources
 For development or testing using web requests, check out the documentation at [docs/web_resources.md](./docs/web_resources.md).
@@ -124,7 +127,7 @@ Suppose the batch contains `n` certificates, and certificate `i` contains recipi
 
 The root of the Merkle tree, which is a 256-bit hash, is issued on the Bitcoin blockchain. The complete Bitcoin transaction outputs are described in 'Transaction structure'.
 
-The Blockchain Certificate given to recipient `i` contains a [2017 Merkle Proof Signature Suite](https://w3c-ccg.github.io/lds-merkleproof2017/)-formatted signature, proving that certificate `i` is contained in the Merkle tree.
+The Blockchain Certificate given to recipient `i` contains a [2019 Merkle Proof Signature Suite](https://w3c-ccg.github.io/lds-merkleproof2019/)-formatted proof, proving that certificate `i` is contained in the Merkle tree.
 
 ![](img/blockchain_certificate_components.png)
 
@@ -145,7 +148,7 @@ These steps establish that the certificate has not been tampered with since it w
 
 ## Hashing a certificate
 
-The Blockchain Certificate JSON contents without the `signature` node is the certificate that the issuer created. This is the value needed to hash for comparison against the receipt. Because there are no guarantees about ordering or formatting of JSON, first canonicalize the certificate (without the `signature`) against the JSON LD schema. This allows us to obtain a deterministic hash across platforms.
+The Blockchain Certificate JSON contents without the `proof` node is the certificate that the issuer created. This is the value needed to hash for comparison against the receipt. Because there are no guarantees about ordering or formatting of JSON, first canonicalize the certificate (without the `proof`) against the JSON LD schema. This allows us to obtain a deterministic hash across platforms.
 
 The detailed steps are described in the [verification process](https://github.com/blockchain-certificates/cert-verifier-js#verification-process).
 
@@ -188,7 +191,7 @@ These steps walk you through issuing in testnet and mainnet mode. Note that the
 
 ## Prerequisites
 
-Decide which chain (Bitcoin or Ethereum) to issue to and follow the steps. The bitcoin chain is currently best supported by the Blockcerts libraries. Follow the steps for the chosen chain.
+Decide which chain (Bitcoin or Ethereum) to issue to and follow the steps. Follow the steps for the chosen chain.
 
 ### Install cert-issuer
 
@@ -208,14 +211,17 @@ python setup.py experimental --blockchain=ethereum
 
 See the docs here for helpful tips on creating / funding blockchain addresses: [docs/testnet_mainnet_addresses](./docs/testnet_mainnet_addresses.md)
 
-
 ## Configuring cert-issuer 
 
-Edit your conf.ini file (the config file for this application).
+Edit your conf.ini file (the config file for this application). See [here](./docs/ethereum_configuration.md) for more details on Ethereum configuration.
+The private key for bitcoin should be the WIF format.
 
 ```
 issuing_address = <issuing-address>
 
+# issuer URL / DID
+verification_method = <verification-method>
+
 chain=<bitcoin_regtest|bitcoin_testnet|bitcoin_mainnet|ethereum_ropsten|ethereum_mainnet|mockchain>
 
 usb_name = </Volumes/path-to-usb/>
@@ -233,7 +239,38 @@ no_safe_mode
 
 Notes:
   - The `bitcoind` option is technically not required in `regtest` mode. `regtest` mode _only_ works with a local bitcoin node. The quick start in docker brushed over this detail by installing a regtest-configured bitcoin node in the docker container.
-  - The Ethereum option does not support a local (test)node currently. The issuer will broadcast the transaction via the Etherscan API.
+  - The Ethereum option does not support a local (test)node currently. The issuer will broadcast the transaction via the Etherscan API or an RPC of their choice.
+
+## Working with DIDs
+To issue and verify a Blockcerts document bound to a DID you need to:
+- generate a DID document referencing the public key source of the issuing address. The verification supports all the DID methods from the [DIF universal resolver](https://resolver.identity.foundation/), but it is recommended you provide your own resolver to the verification library.
+- it is also expected that the DID document contains a `service` property configured similarly to as follows:
+  ```
+    "service": [
+      {
+        "id": "#service-1",
+        "type": "IssuerProfile",
+        "serviceEndpoint": "https://www.blockcerts.org/samples/3.0/issuer-blockcerts.json"
+      }
+    ]
+  ```
+- reference the DID through the `issuer` property of the document to be issued as Blockcerts. Either directly as a string or as the `id` property of an object:
+  ```
+    "issuer": "did:ion:EiA_Z6LQILbB2zj_eVrqfQ2xDm4HNqeJUw5Kj2Z7bFOOeQ",
+  ```
+  or
+  ```
+    "issuer": {
+      "id": "did:ion:EiA_Z6LQILbB2zj_eVrqfQ2xDm4HNqeJUw5Kj2Z7bFOOeQ",
+      ... /* more custom data here. Note that the data from the distant Issuer Profile has display preference in Blockcerts Verifier */
+    }
+  ```
+- finally add to your `conf.ini` file the `verification_method` property pointing to the public key matching the issuing address:
+    ```
+    verification_method=did:ion:EiA_Z6LQILbB2zj_eVrqfQ2xDm4HNqeJUw5Kj2Z7bFOOeQ#key-1 
+    ```
+
+You may try to see the full example DID document by looking up `did:ion:EiA_Z6LQILbB2zj_eVrqfQ2xDm4HNqeJUw5Kj2Z7bFOOeQ` in the [DIF universal resolver](https://resolver.identity.foundation/). 
 
 ## Issuing
 

cert_issuer/__init__.py

@@ -1 +1 @@
-__version__ = '2.0.29'
+__version__ = '3.0.0'

cert_issuer/blockchain_handlers/bitcoin/__init__.py

@@ -7,7 +7,7 @@
 from cert_issuer.blockchain_handlers.bitcoin.connectors import BitcoinServiceProviderConnector, MockServiceProviderConnector
 from cert_issuer.blockchain_handlers.bitcoin.signer import BitcoinSigner
 from cert_issuer.blockchain_handlers.bitcoin.transaction_handlers import BitcoinTransactionHandler
-from cert_issuer.certificate_handlers import CertificateBatchHandler, CertificateV2Handler, CertificateBatchWebHandler, CertificateWebV2Handler
+from cert_issuer.certificate_handlers import CertificateBatchHandler, CertificateV3Handler, CertificateBatchWebHandler, CertificateWebV3Handler
 from cert_issuer.merkle_tree_generator import MerkleTreeGenerator
 from cert_issuer.models import MockTransactionHandler
 from cert_issuer.signer import FileSecretManager
@@ -50,12 +50,14 @@ def instantiate_blockchain_handlers(app_config, file_mode=True):
 
     if file_mode:
         certificate_batch_handler = CertificateBatchHandler(secret_manager=secret_manager,
-                                                            certificate_handler=CertificateV2Handler(),
-                                                            merkle_tree=MerkleTreeGenerator())
+                                                            certificate_handler=CertificateV3Handler(),
+                                                            merkle_tree=MerkleTreeGenerator(),
+                                                            config=app_config)
     else:
         certificate_batch_handler = CertificateBatchWebHandler(secret_manager=secret_manager,
-                                                        certificate_handler=CertificateWebV2Handler(),
-                                                        merkle_tree=MerkleTreeGenerator())
+                                                               certificate_handler=CertificateWebV3Handler(),
+                                                               merkle_tree=MerkleTreeGenerator(),
+                                                               config=app_config)
     if chain == Chain.mockchain:
         transaction_handler = MockTransactionHandler()
         connector = MockServiceProviderConnector()

cert_issuer/blockchain_handlers/ethereum/__init__.py

@@ -4,7 +4,7 @@
 from cert_core import BlockchainType
 from cert_core import Chain, UnknownChainError
 
-from cert_issuer.certificate_handlers import CertificateBatchHandler, CertificateV2Handler
+from cert_issuer.certificate_handlers import CertificateBatchHandler, CertificateV3Handler
 from cert_issuer.blockchain_handlers.ethereum.connectors import EthereumServiceProviderConnector
 from cert_issuer.blockchain_handlers.ethereum.signer import EthereumSigner
 from cert_issuer.blockchain_handlers.ethereum.transaction_handlers import EthereumTransactionHandler
@@ -54,8 +54,9 @@ def instantiate_blockchain_handlers(app_config):
     chain = app_config.chain
     secret_manager = initialize_signer(app_config)
     certificate_batch_handler = CertificateBatchHandler(secret_manager=secret_manager,
-                                                        certificate_handler=CertificateV2Handler(),
-                                                        merkle_tree=MerkleTreeGenerator())
+                                                        certificate_handler=CertificateV3Handler(),
+                                                        merkle_tree=MerkleTreeGenerator(),
+                                                        config=app_config)
     if chain == Chain.mockchain:
         transaction_handler = MockTransactionHandler()
     # ethereum chains

cert_issuer/blockchain_handlers/ethereum/connectors.py

@@ -55,7 +55,7 @@ def __init__(
         if hasattr(app_config, 'ropsten_rpc_url'):
             self.ropsten_rpc_url = app_config.ropsten_rpc_url
             rop_provider_list.append(EthereumRPCProvider(self.ropsten_rpc_url))
-            rop_provider_list.append(EtherscanBroadcaster('https://api-ropsten.etherscan.io/api', etherscan_api_token))
+        rop_provider_list.append(EtherscanBroadcaster('https://api-ropsten.etherscan.io/api', etherscan_api_token))
         # rop_provider_list.append(MyEtherWalletBroadcaster('https://api.myetherwallet.com/rop', None))
         self.connectors[Chain.ethereum_ropsten] = rop_provider_list
 
@@ -137,8 +137,8 @@ def get_balance(self, address):
         """
         Returns the balance in Wei.
         """
-        logging.info('Getting balance with EthereumRPCProvider')
         response = self.w3.eth.getBalance(account=address, block_identifier="latest")
+        logging.info('Getting balance with EthereumRPCProvider: %s', response)
         return response
 
     def get_address_nonce(self, address):
@@ -156,14 +156,20 @@ def __init__(self, base_url, api_token):
         self.base_url = base_url
         self.api_token = api_token
 
+    def send_request(self, method, url, data=None):
+        headers = {
+            'User-Agent': 'Python-urllib/3.8'
+        }
+        response = requests.request(method, url, data=data, headers=headers)
+        return response
+
     def broadcast_tx(self, tx):
         tx_hex = tx
 
         broadcast_url = self.base_url + '?module=proxy&action=eth_sendRawTransaction'
         if self.api_token:
             broadcast_url += '&apikey=%s' % self.api_token
-        response = requests.post(broadcast_url, data={'hex': tx_hex}, headers={'user-agent':'cert-issuer'})
-
+        response = self.send_request('POST', broadcast_url, {'hex': tx_hex})
         if 'error' in response.json():
             logging.error("Etherscan returned an error: %s", response.json()['error'])
             raise BroadcastError(response.json()['error'])
@@ -186,7 +192,7 @@ def get_balance(self, address):
         broadcast_url += '&tag=pending'
         if self.api_token:
             broadcast_url += '&apikey=%s' % self.api_token
-        response = requests.get(broadcast_url, headers={'user-agent':'cert-issuer'})
+        response = self.send_request('GET', broadcast_url)
         if int(response.status_code) == 200:
             if response.json().get('message', None) == 'NOTOK':
                 raise BroadcastError(response.json().get('result', None))
@@ -205,7 +211,7 @@ def get_address_nonce(self, address):
         broadcast_url += '&tag=pending' # Valid tags are 'earliest', 'latest', and 'pending', the last of which includes both pending and committed transactions.
         if self.api_token:
             broadcast_url += '&apikey=%s' % self.api_token
-        response = requests.get(broadcast_url, headers={'user-agent':'cert-issuer'})
+        response = self.send_request('GET', broadcast_url)
         if int(response.status_code) == 200:
             if response.json().get('message', None) == 'NOTOK':
                 raise BroadcastError(response.json().get('result', None))

cert_issuer/certificate_handlers.py

@@ -9,7 +9,7 @@
 
 from cert_issuer.signer import FinalizableSigner
 
-class CertificateV2Handler(CertificateHandler):
+class CertificateV3Handler(CertificateHandler):
     def get_byte_array_to_issue(self, certificate_metadata):
         certificate_json = self._get_certificate_to_issue(certificate_metadata)
         normalized = normalize_jsonld(certificate_json, detect_unmapped_fields=False)
@@ -22,7 +22,7 @@ def add_proof(self, certificate_metadata, merkle_proof):
         :return:
         """
         certificate_json = self._get_certificate_to_issue(certificate_metadata)
-        certificate_json['signature'] = merkle_proof
+        certificate_json['proof'] = merkle_proof
 
         with open(certificate_metadata.blockchain_cert_file_name, 'w') as out_file:
             out_file.write(json.dumps(certificate_json))
@@ -32,7 +32,7 @@ def _get_certificate_to_issue(self, certificate_metadata):
             certificate_json = json.load(unsigned_cert_file)
         return certificate_json
 
-class CertificateWebV2Handler(CertificateHandler):
+class CertificateWebV3Handler(CertificateHandler):
     def get_byte_array_to_issue(self, certificate_json):
         normalized = normalize_jsonld(certificate_json, detect_unmapped_fields=False)
         return normalized.encode('utf-8')
@@ -49,7 +49,7 @@ def add_proof(self, certificate_json, merkle_proof):
 class CertificateBatchWebHandler(BatchHandler):
     def finish_batch(self, tx_id, chain):
         self.proof = []
-        proof_generator = self.merkle_tree.get_proof_generator(tx_id, chain)
+        proof_generator = self.merkle_tree.get_proof_generator(tx_id, self.config.verification_method, chain)
         for metadata in self.certificates_to_issue:
             proof = next(proof_generator)
             self.proof.append(self.certificate_handler.add_proof(metadata, proof))
@@ -99,7 +99,8 @@ def prepare_batch(self):
 
         # validate batch
         for _, metadata in self.certificates_to_issue.items():
-            self.certificate_handler.validate_certificate(metadata)
+            certificate_json = self.certificate_handler._get_certificate_to_issue(metadata)
+            self.certificate_handler.validate_certificate(certificate_json)
 
         # sign batch
         with FinalizableSigner(self.secret_manager) as signer:
@@ -120,7 +121,7 @@ def get_certificate_generator(self):
             yield data_to_issue
 
     def finish_batch(self, tx_id, chain):
-        proof_generator = self.merkle_tree.get_proof_generator(tx_id, chain)
+        proof_generator = self.merkle_tree.get_proof_generator(tx_id, self.config.verification_method, chain)
         for _, metadata in self.certificates_to_issue.items():
             proof = next(proof_generator)
             self.certificate_handler.add_proof(metadata, proof)

cert_issuer/config.py

@@ -34,6 +34,7 @@ def add_arguments(p):
     p.add('-c', '--my-config', required=False, env_var='CONFIG_FILE',
           is_config_file=True, help='config file path')
     p.add_argument('--issuing_address', required=True, help='issuing address', env_var='ISSUING_ADDRESS')
+    p.add_argument('--verification_method', required=True, help='Verification method for the Linked Data Proof', env_var='VERIFICATION_METHOD')
     p.add_argument('--usb_name', required=True, help='usb path to key_file', env_var='USB_NAME')
     p.add_argument('--key_file', required=True,
                    help='name of file on USB containing private key', env_var='KEY_FILE')

cert_issuer/helpers.py

@@ -100,3 +100,21 @@ def to_pycoin_chain(chain):
         return 'BTC'
     else:
         raise UnknownChainError(chain.name)
+
+def tx_to_blink(chain, tx_id):
+    blink = 'blink:'
+    if chain == Chain.bitcoin_regtest:
+        blink += 'btc:regtest:'
+    elif chain == Chain.bitcoin_testnet:
+        blink += 'btc:testnet:'
+    elif chain == Chain.bitcoin_mainnet:
+        blink += 'btc:mainnet:'
+    elif chain == Chain.ethereum_ropsten:
+        blink += 'eth:ropsten:'
+    elif chain == Chain.ethereum_mainnet:
+        blink += 'eth:mainnet:'
+    elif chain == Chain.mockchain:
+        blink += 'mocknet:'
+    else:
+        raise UnknownChainError(chain.name)
+    return blink + tx_id

cert_issuer/merkle_tree_generator.py

@@ -1,8 +1,12 @@
 import hashlib
+import logging
+from datetime import datetime
 
 from cert_core import Chain
 from merkletools import MerkleTools
 from pycoin.serialize import h2b
+from lds_merkle_proof_2019.merkle_proof_2019 import MerkleProof2019
+from cert_issuer import helpers
 
 
 def hash_byte_array(data):
@@ -40,7 +44,7 @@ def get_blockchain_data(self):
         merkle_root = self.tree.get_merkle_root()
         return h2b(ensure_string(merkle_root))
 
-    def get_proof_generator(self, tx_id, chain=Chain.bitcoin_mainnet):
+    def get_proof_generator(self, tx_id, verification_method, chain=Chain.bitcoin_mainnet):
         """
         Returns a generator (1-time iterator) of proofs in insertion order.
 
@@ -59,16 +63,25 @@ def get_proof_generator(self, tx_id, chain=Chain.bitcoin_mainnet):
                     dict2[key] = ensure_string(value)
                 proof2.append(dict2)
             target_hash = ensure_string(self.tree.get_leaf(index))
+            mp2019 = MerkleProof2019()
+            merkle_json = {
+                  "path": proof2,
+                  "merkleRoot": root,
+                  "targetHash": target_hash,
+                  "anchors": [
+                    helpers.tx_to_blink(chain, tx_id)
+                  ]
+                }
+            logging.info('merkle_json: %s', str(merkle_json))
+
+            proof_value = mp2019.encode(merkle_json)
             merkle_proof = {
-                "type": ['MerkleProof2017', 'Extension'],
-                "merkleRoot": root,
-                "targetHash": target_hash,
-                "proof": proof2,
-                "anchors": [{
-                    "sourceId": to_source_id(tx_id, chain),
-                    "type": chain.blockchain_type.external_display_value,
-                    "chain": chain.external_display_value
-                }]}
+                "type": "MerkleProof2019",
+                "created": datetime.now().isoformat(),
+                "proofValue": proof_value.decode('utf8'),
+                "proofPurpose": "assertionMethod",
+                "verificationMethod": verification_method
+            }
             yield merkle_proof