Merge branch 'release/v1.2'

.travis.yml

@@ -13,20 +13,14 @@ script:
   - pytest test_bbc_app.py
   - pytest test_bbc_app_multi_core.py
   - pytest test_bbc_app_multi_core_many_transactions.py
-  - pytest test_bbc_app_multi_core_many_transactions_bson.py
   - pytest test_bbc_app_multi_core_notification_with_cancelation.py
   - pytest test_bbc_app_multi_core_notification_with_disconnection.py
   - pytest test_bbc_app_multiclient.py
   - pytest test_bbc_app_multiclient_traverse.py
-  - pytest test_bbc_app_multiclient_traverse_bson.py
-  - pytest test_bbc_app_multi_core_domain_key.py
   - pytest test_bbc_app_admin_tools.py
   - pytest test_bbc_app_multi_core_compromising_data.py
   - pytest test_bbc_app_multi_core_anycast.py
   - pytest test_domain_info_adv.py
-  - pytest test_cross_ref.py
-  - pytest test_cross_ref_bson.py
-  - pytest test_cross_ref_multi_domain0nodes.py
   - pytest test_fileproof_multi_core.py
   - pytest test_bbc_config.py
   - pytest test_user_message_routing.py
@@ -39,3 +33,8 @@ script:
   - pytest test_pending_request.py
   - pytest test_bbc_ping.py
   - pytest test_bbc_stats.py
+  - cd compat
+  - pytest test_bbc_app_multi_core_many_transactions_bson.py
+  - pytest test_bbc_app_multiclient_traverse_bson.py
+  - pytest test_bbc_app_multi_core_compat.py
+  - pytest test_bbclib_bson.py

MANIFEST.in

@@ -7,8 +7,6 @@ include package.json
 include *.rst
 include prepare.sh
 include prepare.py
-recursive-include libs/openssl *
-recursive-include libs/libbbcsig *
 graft django
 prune django/contrib/admin/bin
 graft docs

README.md

@@ -1,6 +1,7 @@
 Core system of BBc-1 (Beyond Blockchain One)
 ===========================================
 [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![Build Status](https://travis-ci.org/beyond-blockchain/bbc1.svg?branch=develop)](https://travis-ci.org/beyond-blockchain/bbc1)
 
 This project is a Python-based reference implementation of BBc-1, a trustable system of record keeping beyond blockchains.
 
@@ -11,15 +12,21 @@ A transaction should be signed by the players who are the stake holders of the d
 BBc-1 can be applied to both private/enterprise use and public use. BBc-1 has a concept of *domain* for determining a region of data management. Any networking implementation (like Kademlia for P2P topology management) can be applied for each domain.
 Although there are many TODOs in BBc-1, this reference implementation includes most of the concept of BBc-1 and would work in private/enterprise systems. When sophisticated P2P algorithms are ready, BBc-1 will be able to support public use cases.
 
-The source codes in this repository is a platform of BBc-1 and bbc\_core.py is the main process of a core node. 
+The source codes in this repository is a platform of BBc-1 and bbc\_core.py is the main process of a core node.
 The APIs of BBc-1 is defined in bbc\_app.py and bbclib.py. So application developers should import them in your apps.
- For building a BBc-1 system, bbc1 package needs to be installed in the hosts and you need to run bbc\_core.py on 
+ For building a BBc-1 system, bbc1 package needs to be installed in the hosts and you need to run bbc\_core.py on
  each host. In order to configure the BBc-1 network, the utilities in utils/ directory are available. They are a kind
   of BBc-1 application, so that you can develop your own management tools.
- 
+
 
 For the details, please read documents in docs/ directory. Not only documents but slide decks (PDF) explain the design of the BBc-1 and its implementation.
 
+## Recent changes regarding bbclib.py
+
+bbclib.py (and the data format) is redesigned, resulting in splitting it to two parts; bbclib_core.py and bbclib_wire.py. bbclib.py is just a wrapper for those two scripts. The way of serializing BBc-1 transaction object and deserializing transaction data will change in v1.2.
+
+Please see [here](docs/BBc1_data_format_ja.md) in detail.
+
 ## Documents
 Some documents are available in docs/.
 * Policy, design and analysis
@@ -34,12 +41,13 @@ Some documents are available in docs/.
     * [how_to_use_in_nat_environment.md](docs/how_to_use_in_nat_environment.md)
     * [libbbcsig_dll_build_for_Windows_x64_ja.md](docs/libbbcsig_dll_build_for_Windows_x64_ja.md)
 * Programing
-    * [BBc1_programming_guide_v1.0_ja.md](docs/BBc1_programming_guide_v1.0_ja.md)
+    * [BBc1_programming_guide_v1.2_ja.md](docs/BBc1_programming_guide_v1.2_ja.md)
     * [BBc1_core_tutorial_file_proof_ja.md](docs/BBc1_core_tutorial_file_proof_ja.md)
+    * [BBc1_data_format_ja.md](docs/BBc1_data_format_ja.md)
 * API reference (Coming soon. Currently, something wrong in building docs)
     * [https://bbc-1.readthedocs.io/en/latest/](https://bbc-1.readthedocs.io/en/latest/)
     * You can read API docs in your local host by the following command:
-        ```python
+        ```shell
         cd docs/api/_build/html
         pipenv run python -m http.server
         ```
@@ -52,7 +60,7 @@ Some documents are available in docs/.
     - virtualenv is recommended
         - ```python -mvenv venv```
     - In some environment, [pipenv](https://docs.pipenv.org) does not work well.
-        - Some bugs seems to be in the installation scripts. So, please do not use pipenv now. 
+        - Some bugs seems to be in the installation scripts. So, please do not use pipenv now.
 
 * tools for macOS by Homebrew
     ```
@@ -81,8 +89,8 @@ Some documents are available in docs/.
     python -mvenv venv
     source venv/bin/activate
     pip install -r requirements.txt
-    ``` 
-    
+    ```
+
 6. Start bbc_core.py on a terminal
     ```
     cd core
@@ -149,8 +157,6 @@ The working directory of BBc-1 on the docker container is mounted on docker/data
     - docker environments
 * tests/
     - test codes for pytest
-* libs/
-    - some libraries used by BBc-1
 * docs/
     - docs about BBc-1 and its reference implementation
 * somewhere/.bbc1/

README.rst

@@ -19,6 +19,12 @@ The APIs of BBc-1 is defined in bbc\_app.py and bbclib.py. So application develo
 
 For the details, please read documents in docs/ directory. Not only documents but slide decks (PDF) explain the design of the BBc-1 and its implementation.
 
+## Recent changes regarding bbclib.py
+
+bbclib.py (and the data format) is redesigned, resulting in splitting it to two parts; bbclib_core.py and bbclib_wire.py. bbclib.py is just a wrapper for those two scripts. The way of serializing BBc-1 transaction object and deserializing transaction data will change in v1.2.
+
+Please see [here](docs/BBc1_data_format_ja.md) in detail.
+
 ## Documents
 Some documents are available in docs/.
 * Policy, design and analysis
@@ -148,8 +154,6 @@ The working directory of BBc-1 on the docker container is mounted on docker/data
     - docker environments
 * tests/
     - test codes for pytest
-* libs/
-    - some libraries used by BBc-1
 * docs/
     - docs about BBc-1 and its reference implementation
 * somewhere/.bbc1/

bbc1/core/bbc_app.py

@@ -503,14 +503,13 @@ def gather_signatures(self, txobj, reference_obj=None, asset_files=None, destina
         if reference_obj is None and destinations is None:
             return False
         dat = self._make_message_structure(MsgType.REQUEST_GATHER_SIGNATURE)
-        dat[KeyType.transaction_data] = txobj.serialize()
+        dat[KeyType.transaction_data] = bbclib.serialize(txobj)
         dat[KeyType.transaction_id] = txobj.transaction_id
         if anycast:
             dat[KeyType.is_anycast] = True
         if reference_obj is not None:
             dat[KeyType.destination_user_ids] = reference_obj.get_destinations()
-            referred_transactions = dict()
-            referred_transactions.update(reference_obj.get_referred_transaction())
+            referred_transactions = {reference_obj.transaction_id: bbclib.serialize(reference_obj.ref_transaction)}
             if len(referred_transactions) > 0:
                 dat[KeyType.transactions] = referred_transactions
         elif destinations is not None:
@@ -537,12 +536,7 @@ def sendback_signature(self, dest_user_id=None, transaction_id=None, ref_index=-
         dat[KeyType.destination_user_id] = dest_user_id
         dat[KeyType.transaction_id] = transaction_id
         dat[KeyType.ref_index] = ref_index
-        if signature.format_type in [bbclib.BBcFormat.FORMAT_BSON, bbclib.BBcFormat.FORMAT_BSON_COMPRESS_BZ2]:
-            dat[KeyType.signature] = bson.dumps(signature.serialize())
-            dat[KeyType.transaction_data_format] = bbclib.BBcFormat.FORMAT_BSON
-        else:
-            dat[KeyType.signature] = signature.serialize()
-            dat[KeyType.transaction_data_format] = bbclib.BBcFormat.FORMAT_BINARY
+        dat[KeyType.signature] = signature.pack()
         if query_id is not None:
             dat[KeyType.query_id] = query_id
         return self._send_msg(dat)
@@ -569,26 +563,26 @@ def sendback_denial_of_sign(self, dest_user_id=None, transaction_id=None, reason
             dat[KeyType.query_id] = query_id
         return self._send_msg(dat)
 
-    def insert_transaction(self, tx_obj):
+    def insert_transaction(self, txobj):
         """Request to insert a legitimate transaction
 
         Args:
-            tx_obj (BBcTransaction): Transaction object to insert
+            txobj (BBcTransaction): Transaction object to insert
         Returns:
             bytes: query_id
         """
-        if tx_obj.transaction_id is None:
-            tx_obj.digest()
+        if txobj.transaction_id is None:
+            txobj.digest()
         dat = self._make_message_structure(MsgType.REQUEST_INSERT)
-        dat[KeyType.transaction_data] = tx_obj.serialize()
+        dat[KeyType.transaction_data] = bbclib.serialize(txobj)
         ast = dict()
-        for evt in tx_obj.events:
+        for evt in txobj.events:
             if evt.asset is None:
                 continue
             asset_digest, content = evt.asset.get_asset_file()
             if content is not None:
                 ast[evt.asset.asset_id] = content
-        for rtn in tx_obj.relations:
+        for rtn in txobj.relations:
             if rtn.asset is None:
                 continue
             asset_digest, content = rtn.asset.get_asset_file()
@@ -819,7 +813,6 @@ def include_cross_ref(self, txobj):
         """
         if len(self.cross_ref_list) > 0:
             txobj.add(cross_ref=self.cross_ref_list.pop(0))
-            txobj.cross_ref.format_type = txobj.format_type
 
 
 class Callback:
@@ -990,12 +983,7 @@ def proc_resp_gather_signature(self, dat):
         if KeyType.status not in dat or dat[KeyType.status] < ESUCCESS:
             self.queue.put(dat)
             return
-        format_type = dat[KeyType.transaction_data_format]
-        if format_type in [bbclib.BBcFormat.FORMAT_BSON, bbclib.BBcFormat.FORMAT_BSON_COMPRESS_BZ2]:
-            sigdata = bson.loads(dat[KeyType.signature])
-        else:
-            sigdata = dat[KeyType.signature]
-        sig = bbclib.recover_signature_object(sigdata, format_type=format_type)
+        sig = bbclib.recover_signature_object(dat[KeyType.signature])
         self.queue.put({KeyType.status: ESUCCESS, KeyType.result: (dat[KeyType.ref_index], dat[KeyType.source_user_id], sig)})
 
     def proc_resp_insert(self, dat):

bbc1/core/bbc_core.py

@@ -37,8 +37,8 @@
 import sys
 sys.path.extend(["../../"])
 from bbc1.core import bbclib
+from bbc1.core.bbclib import MsgType
 from bbc1.core.message_key_types import KeyType, to_2byte
-from bbc1.core.bbclib import BBcTransaction, MsgType
 from bbc1.core import bbc_network, user_message_routing, data_handler, repair_manager, message_key_types, logger
 from bbc1.core import domain0_manager, query_management, bbc_stats
 from bbc1.core.bbc_config import BBcConfig
@@ -105,15 +105,16 @@ def _create_search_result(txobj_dict, asset_files_dict):
     """Create transaction search result"""
     response_info = dict()
     for txid, txobj in txobj_dict.items():
+        txdata = bbclib.serialize(txobj)
         if txid != txobj.transaction_id:
-            response_info.setdefault(KeyType.compromised_transactions, list()).append(txobj.transaction_data)
+            response_info.setdefault(KeyType.compromised_transactions, list()).append(txdata)
             response_info.setdefault(KeyType.compromised_transaction_ids, list()).append(txid)
             continue
         txobj_is_valid, valid_assets, invalid_assets = bbclib.validate_transaction_object(txobj, asset_files_dict)
         if txobj_is_valid:
-            response_info.setdefault(KeyType.transactions, list()).append(txobj.transaction_data)
+            response_info.setdefault(KeyType.transactions, list()).append(txdata)
         else:
-            response_info.setdefault(KeyType.compromised_transactions, list()).append(txobj.transaction_data)
+            response_info.setdefault(KeyType.compromised_transactions, list()).append(txdata)
             response_info.setdefault(KeyType.compromised_transaction_ids, list()).append(txid)
 
         if len(valid_assets) > 0:
@@ -151,7 +152,6 @@ def __init__(self, p2p_port=None, core_port=None, use_domain0=False, ip4addr=Non
         self.logger.debug("config = %s" % conf)
         self.search_max_count = conf['search_config']['max_count']
         self.traverse_max_count = conf['search_config']['max_traverse']
-        self.test_tx_obj = BBcTransaction()
         self.insert_notification_user_list = dict()
         self.networking = bbc_network.BBcNetwork(self.config, core=self, p2p_port=p2p_port,
                                                  external_ip4addr=ip4addr, external_ip6addr=ip6addr,
@@ -450,7 +450,9 @@ def _process(self, socket, dat, payload_type):
             retmsg = _make_message_structure(domain_id, MsgType.RESPONSE_GATHER_SIGNATURE,
                                              dat[KeyType.destination_user_id], dat[KeyType.query_id])
             if KeyType.signature in dat:
-                retmsg[KeyType.transaction_data_format] = dat[KeyType.transaction_data_format]
+                if KeyType.transaction_data_format in dat:
+                    # -- for backward compatibility
+                    retmsg[KeyType.transaction_data_format] = dat[KeyType.transaction_data_format]
                 retmsg[KeyType.signature] = dat[KeyType.signature]
                 retmsg[KeyType.ref_index] = dat[KeyType.ref_index]
             elif KeyType.status not in dat:
@@ -768,12 +770,13 @@ def validate_transaction(self, txdata, asset_files=None):
             asset_files (dict): dictionary of {asset_id: content} for the transaction
         Returns:
             BBcTransaction: if validation fails, None returns.
+            int (short): 2-byte value of BBcFormat type or None
         """
-        txobj = BBcTransaction()
-        if not txobj.deserialize(txdata):
+        txobj, fmt_type = bbclib.deserialize(txdata)
+        if not txobj:
             self.stats.update_stats_increment("transaction", "invalid", 1)
             self.logger.error("Fail to deserialize transaction data")
-            return None
+            return None, None
         txobj.digest()
 
         txobj_is_valid, valid_assets, invalid_assets = bbclib.validate_transaction_object(txobj, asset_files)
@@ -783,9 +786,9 @@ def validate_transaction(self, txdata, asset_files=None):
             self.stats.update_stats_increment("asset_file", "invalid", 1)
 
         if txobj_is_valid and len(invalid_assets) == 0:
-            return txobj
+            return txobj, fmt_type
         else:
-            return None
+            return None, None
 
     def insert_transaction(self, domain_id, txdata, asset_files):
         """Insert transaction into ledger
@@ -806,15 +809,15 @@ def insert_transaction(self, domain_id, txdata, asset_files):
             self.stats.update_stats_increment("transaction", "insert_fail_count", 1)
             self.logger.error("Insert is not allowed in domain_global_0")
             return "Insert is not allowed in domain_global_0"
-        txobj = self.validate_transaction(txdata, asset_files)
+        txobj, fmt_type = self.validate_transaction(txdata, asset_files)
         if txobj is None:
             self.stats.update_stats_increment("transaction", "insert_fail_count", 1)
             self.logger.error("Bad transaction format")
             return "Bad transaction format"
         self.logger.debug("[node:%s] insert_transaction %s" %
                           (self.networking.domains[domain_id]['name'], binascii.b2a_hex(txobj.transaction_id[:4])))
 
-        asset_group_ids = self.networking.domains[domain_id]['data'].insert_transaction(txdata, txobj=txobj,
+        asset_group_ids = self.networking.domains[domain_id]['data'].insert_transaction(txdata, txobj=txobj, fmt_type=fmt_type,
                                                                                         asset_files=asset_files)
         if asset_group_ids is None:
             self.stats.update_stats_increment("transaction", "insert_fail_count", 1)
@@ -1040,7 +1043,7 @@ def _traverse_transactions(self, domain_id, transaction_id, asset_group_id=None,
                             break
                     if not flag:
                         continue
-                tx_brothers.append(ret_txobj[txid].transaction_data)
+                tx_brothers.append(bbclib.serialize(ret_txobj[txid]))
                 if len(ret_asset_files) > 0:
                     asset_files.update(ret_asset_files)
 

bbc1/core/bbc_network.py

@@ -40,7 +40,6 @@
 from bbc1.core import query_management, message_key_types, logger
 from bbc1.core import bbclib
 from bbc1.core.message_key_types import to_2byte, PayloadType, KeyType, InfraMessageCategory
-from bbc1.core.bbc_error import *
 
 TCP_THRESHOLD_SIZE = 1300
 ZEROS = bytes([0] * 32)