This tutorial will describe the steps required to use the CouchDB as the state database with Hyperledger Fabric. By now, you should be familiar with Fabric concepts and have explored some of the samples and tutorials.
The tutorial will take you through the following steps:
cdb-enable-couch
cdb-create-index
cdb-add-index
cdb-install-instantiate
cdb-query
cdb-pagination
cdb-update-index
cdb-delete-index
For a deeper dive into CouchDB refer to couchdb_as_state_database
and for more information on the Fabric ledger refer to the Ledger topic. Follow the tutorial below for details on how to leverage CouchDB in your blockchain network.
Throughout this tutorial we will use the Marbles sample as our use case to demonstrate how to use CouchDB with Fabric and will deploy Marbles to the build_network
(BYFN) tutorial network. You should have completed the task install
. However, running the BYFN tutorial is not a prerequisite for this tutorial, instead the necessary commands are provided throughout this tutorial to use the network.
Fabric supports two types of peer databases. LevelDB is the default state database embedded in the peer node and stores chaincode data as simple key-value pairs and supports key, key range, and composite key queries only. CouchDB is an optional alternate state database that supports rich queries when chaincode data values are modeled as JSON. Rich queries are more flexible and efficient against large indexed data stores, when you want to query the actual data value content rather than the keys. CouchDB is a JSON document datastore rather than a pure key-value store therefore enabling indexing of the contents of the documents in the database.
In order to leverage the benefits of CouchDB, namely content-based JSON queries,your data must be modeled in JSON format. You must decide whether to use LevelDB or CouchDB before setting up your network. Switching a peer from using LevelDB to CouchDB is not supported due to data compatibility issues. All peers on the network must use the same database type. If you have a mix of JSON and binary data values, you can still use CouchDB, however the binary values can only be queried based on key, key range, and composite key queries.
CouchDB runs as a separate database process alongside the peer, therefore there are additional considerations in terms of setup, management, and operations. A docker image of CouchDB is available and we recommend that it be run on the same server as the peer. You will need to setup one CouchDB container per peer and update each peer container by changing the configuration found in core.yaml
to point to the CouchDB container. The core.yaml
file must be located in the directory specified by the environment variable FABRIC_CFG_PATH:
- For docker deployments,
core.yaml
is pre-configured and located in the peer containerFABRIC_CFG_PATH
folder. However when using docker environments, you typically pass environment variables by editing thedocker-compose-couch.yaml
to override the core.yaml - For native binary deployments,
core.yaml
is included with the release artifact distribution.
Edit the stateDatabase
section of core.yaml
. Specify CouchDB
as the stateDatabase
and fill in the associated couchDBConfig
properties. For more details on configuring CouchDB to work with fabric, refer here. To view an example of a core.yaml file configured for CouchDB, examine the BYFN docker-compose-couch.yaml
in the HyperLedger/fabric-samples/first-network
directory.
Why are indexes important?
Indexes allow a database to be queried without having to examine every row with every query, making them run faster and more efficiently. Normally, indexes are built for frequently occurring query criteria allowing the data to be queried more efficiently. To leverage the major benefit of CouchDB -- the ability to perform rich queries against JSON data -- indexes are not required, but they are strongly recommended for performance. Also, if sorting is required in a query, CouchDB requires an index of the sorted fields.
Note
Rich queries that do not have an index will work but may throw a warning in the CouchDB log that the index was not found. However, if a rich query includes a sort specification, then an index on that field is required; otherwise, the query will fail and an error will be thrown.
To demonstrate building an index, we will use the data from the Marbles sample. In this example, the Marbles data structure is defined as:
type marble struct {
ObjectType string `json:"docType"` //docType is used to distinguish the various types of objects in state database
Name string `json:"name"` //the field tags are needed to keep case from bouncing around
Color string `json:"color"`
Size int `json:"size"`
Owner string `json:"owner"`
}
In this structure, the attributes (docType
, name
, color
, size
, owner
) define the ledger data associated with the asset. The attribute docType
is a pattern used in the chaincode to differentiate different data types that may need to be queried separately. When using CouchDB, it recommended to include this docType
attribute to distinguish each type of document in the chaincode namespace. (Each chaincode is represented as its own CouchDB database, that is, each chaincode has its own namespace for keys.)
With respect to the Marbles data structure, docType
is used to identify that this document/asset is a marble asset. Potentially there could be other documents/assets in the chaincode database. The documents in the database are searchable against all of these attribute values.
When defining an index for use in chaincode queries, each one must be defined in its own text file with the extension *.json and the index definition must be formatted in the CouchDB index JSON format.
To define an index, three pieces of information are required:
- `fields`: these are the frequently queried fields
- `name`: name of the index
- `type`: always json in this context
For example, a simple index named foo-index
for a field named foo
.
{
"index": {
"fields": ["foo"]
},
"name" : "foo-index",
"type" : "json"
}
Optionally the design document attribute ddoc
can be specified on the index definition. A design document is CouchDB construct designed to contain indexes. Indexes can be grouped into design documents for efficiency but CouchDB recommends one index per design document.
Tip
When defining an index it is a good practice to include the ddoc
attribute and value along with the index name. It is important to include this attribute to ensure that you can update the index later if needed. Also it gives you the ability to explicitly specify which index to use on a query.
Here is another example of an index definition from the Marbles sample with the index name indexOwner
using multiple fields docType
and owner
and includes the ddoc
attribute:
{
"index":{
"fields":["docType","owner"] // Names of the fields to be queried
},
"ddoc":"indexOwnerDoc", // (optional) Name of the design document in which the index will be created.
"name":"indexOwner",
"type":"json"
}
In the example above, if the design document indexOwnerDoc
does not already exist, it is automatically created when the index is deployed. An index can be constructed with one or more attributes specified in the list of fields and any combination of attributes can be specified. An attribute can exist in multiple indexes for the same docType. In the following example, index1
only includes the attribute owner
, index2
includes the attributes owner and color
and index3
includes the attributes owner, color and size
. Also, notice each index definition has its own ddoc
value, following the CouchDB recommended practice.
{
"index":{
"fields":["owner"] // Names of the fields to be queried
},
"ddoc":"index1Doc", // (optional) Name of the design document in which the index will be created.
"name":"index1",
"type":"json"
}
{
"index":{
"fields":["owner", "color"] // Names of the fields to be queried
},
"ddoc":"index2Doc", // (optional) Name of the design document in which the index will be created.
"name":"index2",
"type":"json"
}
{
"index":{
"fields":["owner", "color", "size"] // Names of the fields to be queried
},
"ddoc":"index3Doc", // (optional) Name of the design document in which the index will be created.
"name":"index3",
"type":"json"
}
In general, you should model index fields to match the fields that will be used in query filters and sorts. For more details on building an index in JSON format refer to the CouchDB documentation.
A final word on indexing, Fabric takes care of indexing the documents in the database using a pattern called index warming
. CouchDB does not typically index new or updated documents until the next query. Fabric ensures that indexes stay 'warm' by requesting an index update after every block of data is committed. This ensures queries are fast because they do not have to index documents before running the query. This process keeps the index current and refreshed every time new records are added to the state database.
Once you finalize an index, it is ready to be packaged with your chaincode for deployment by being placed alongside it in the appropriate metadata folder.
If your chaincode installation and instantiation uses the Hyperledger Fabric Node SDK, the JSON index files can be located in any folder as long as it conforms to this directory structure. During the chaincode installation using the client.installChaincode() API, include the attribute (metadataPath
) in the installation request. The value of the metadataPath is a string representing the absolute path to the directory structure containing the JSON index file(s).
Alternatively, if you are using the peer-commands
to install and instantiate the chaincode, then the JSON index files must be located under the path META-INF/statedb/couchdb/indexes
which is located inside the directory where the chaincode resides.
The Marbles sample below illustrates how the index is packaged with the chaincode which will be installed using the peer commands.
Try it yourself
Before installing and instantiating the marbles chaincode, we need to start up the BYFN network. For the sake of this tutorial, we want to operate from a known initial state. The following command will kill any active or stale docker containers and remove previously generated artifacts. Therefore let's run the following command to clean up any previous environments:
cd fabric-samples/first-network ./byfn.sh down
Now start up the BYFN network with CouchDB by running the following command:
./byfn.sh up -c mychannel -s couchdbThis will create a simple Fabric network consisting of a single channel named mychannel with two organizations (each maintaining two peer nodes) and an ordering service while using CouchDB as the state database.
Client applications interact with the blockchain ledger through chaincode. As such we need to install the chaincode on every peer that will execute and endorse our transactions and instantiate the chaincode on the channel. In the previous section, we demonstrated how to package the chaincode so they should be ready for deployment.
Chaincode is installed onto a peer and then instantiated onto the channel using peer-commands
.
- Use the peer chaincode install command to install the Marbles chaincode on a peer.
Try it yourself
Assuming you have started the BYFN network, navigate into the CLI container using the command:
docker exec -it cli bash
Use the following command to install the Marbles chaincode from the git repository onto a peer in your BYFN network. The CLI container defaults to using peer0 of org1:
peer chaincode install -n marbles -v 1.0 -p github.com/chaincode/marbles02/go
2. Issue the peer chaincode instantiate command to instantiate the chaincode on a channel.
Try it yourself
To instantiate the Marbles sample on the BYFN channel
mychannel
run the following command:export CHANNEL_NAME=mychannel peer chaincode instantiate -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C $CHANNEL_NAME -n marbles -v 1.0 -c '{"Args":["init"]}' -P "OR ('Org0MSP.peer','Org1MSP.peer')"
Indexes will be deployed to each peer's CouchDB state database once the chaincode is both installed on the peer and instantiated on the channel. You can verify that the CouchDB index was created successfully by examining the peer log in the Docker container.
Try it yourself
To view the logs in the peer docker container, open a new Terminal window and run the following command to grep for message confirmation that the index was created.
docker logs peer0.org1.example.com 2>&1 | grep "CouchDB index"
You should see a result that looks like the following:
[couchdb] CreateIndex -> INFO 0be Created CouchDB index [indexOwner] in state database [mychannel_marbles] using design document [_design/indexOwnerDoc]
Note
If Marbles was not installed on the BYFN peer
peer0.org1.example.com
, you may need to replace it with the name of a different peer where Marbles was installed.
Now that the index has been defined in the JSON file and deployed alongside the chaincode, chaincode functions can execute JSON queries against the CouchDB state database, and thereby peer commands can invoke the chaincode functions.
Specifying an index name on a query is optional. If not specified, and an index already exists for the fields being queried, the existing index will be automatically used.
Tip
It is a good practice to explicitly include an index name on a query using the use_index
keyword. Without it, CouchDB may pick a less optimal index. Also CouchDB may not use an index at all and you may not realize it, at the low volumes during testing. Only upon higher volumes you may realize slow performance because CouchDB is not using an index and you assumed it was.
You can perform complex rich queries against the chaincode data values using the CouchDB JSON query language within chaincode. As we explored above, the marbles02 sample chaincode includes an index and rich queries are defined in the functions - queryMarbles
and queryMarblesByOwner
:
queryMarbles --
Example of an ad hoc rich query. This is a query where a (selector) string can be passed into the function. This query would be useful to client applications that need to dynamically build their own selectors at runtime. For more information on selectors refer to CouchDB selector syntax.
queryMarblesByOwner --
Example of a parameterized query where the query logic is baked into the chaincode. In this case the function accepts a single argument, the marble owner. It then queries the state database for JSON documents matching the docType of “marble” and the owner id using the JSON query syntax.
In absence of a client application to test rich queries defined in chaincode, peer commands can be used. Peer commands run from the command line inside the docker container. We will customize the peer chaincode query command to use the Marbles index indexOwner
and query for all marbles owned by "tom" using the queryMarbles
function.
Try it yourself
Before querying the database, we should add some data. Run the following command in the peer container to create a marble owned by "tom":
peer chaincode invoke -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C $CHANNEL_NAME -n marbles -c '{"Args":["initMarble","marble1","blue","35","tom"]}'After an index has been deployed during chaincode instantiation, it will automatically be utilized by chaincode queries. CouchDB can determine which index to use based on the fields being queried. If an index exists for the query criteria it will be used. However the recommended approach is to specify the
use_index
keyword on the query. The peer command below is an example of how to specify the index explicitly in the selector syntax by including theuse_index
keyword:// Rich Query with index name explicitly specified: peer chaincode query -C $CHANNEL_NAME -n marbles -c '{"Args":["queryMarbles", "{\"selector\":{\"docType\":\"marble\",\"owner\":\"tom\"}, \"use_index\":[\"_design/indexOwnerDoc\", \"indexOwner\"]}"]}'
Delving into the query command above, there are three arguments of interest:
- *
queryMarbles
Name of the function in the Marbles chaincode. Notice a shim
shim.ChaincodeStubInterface
is used to access and modify the ledger. ThegetQueryResultForQueryString()
passes the queryString to the shim APIgetQueryResult()
.
func (t *SimpleChaincode) queryMarbles(stub shim.ChaincodeStubInterface, args []string) pb.Response {
// 0
// "queryString"
if len(args) < 1 {
return shim.Error("Incorrect number of arguments. Expecting 1")
}
queryString := args[0]
queryResults, err := getQueryResultForQueryString(stub, queryString)
if err != nil {
return shim.Error(err.Error())
}
return shim.Success(queryResults)
}
- *
{"selector":{"docType":"marble","owner":"tom"}
This is an example of an ad hoc selector string which finds all documents of type
marble
where theowner
attribute has a value oftom
.- *
"use_index":["_design/indexOwnerDoc", "indexOwner"]
Specifies both the design doc name
indexOwnerDoc
and index nameindexOwner
. In this example the selector query explicitly includes the index name, specified by using theuse_index
keyword. Recalling the index definition abovecdb-create-index
, it contains a design doc,"ddoc":"indexOwnerDoc"
. With CouchDB, if you plan to explicitly include the index name on the query, then the index definition must include theddoc
value, so it can be referenced with theuse_index
keyword.
The query runs successfully and the index is leveraged with the following results:
Query Result: [{"Key":"marble1", "Record":{"color":"blue","docType":"marble","name":"marble1","owner":"tom","size":35}}]
When large result sets are returned by CouchDB queries, a set of APIs is available which can be called by chaincode to paginate the list of results. Pagination provides a mechanism to partition the result set by specifying a pagesize
and a start point -- a bookmark
which indicates where to begin the result set. The client application iteratively invokes the chaincode that executes the query until no more results are returned. For more information refer to this topic on pagination with CouchDB.
We will use the Marbles sample function queryMarblesWithPagination
to demonstrate how pagination can be implemented in chaincode and the client application.
queryMarblesWithPagination --
Example of an ad hoc rich query with pagination. This is a query where a (selector) string can be passed into the function similar to the above example. In this case, a
pageSize
is also included with the query as well as abookmark
.
In order to demonstrate pagination, more data is required. This example assumes that you have already added marble1 from above. Run the following commands in the peer container to create four more marbles owned by "tom", to create a total of five marbles owned by "tom":
Try it yourself
peer chaincode invoke -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C $CHANNEL_NAME -n marbles -c '{"Args":["initMarble","marble2","yellow","35","tom"]}'
peer chaincode invoke -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C $CHANNEL_NAME -n marbles -c '{"Args":["initMarble","marble3","green","20","tom"]}'
peer chaincode invoke -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C $CHANNEL_NAME -n marbles -c '{"Args":["initMarble","marble4","purple","20","tom"]}'
peer chaincode invoke -o orderer.example.com:7050 --tls --cafile /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem -C $CHANNEL_NAME -n marbles -c '{"Args":["initMarble","marble5","blue","40","tom"]}'
In addition to the arguments for the query in the previous example, queryMarblesWithPagination adds pagesize
and bookmark
. PageSize
specifies the number of records to return per query. The bookmark
is an "anchor" telling couchDB where to begin the page. (Each page of results returns a unique bookmark.)
- *
queryMarblesWithPagination
Name of the function in the Marbles chaincode. Notice a shim
shim.ChaincodeStubInterface
is used to access and modify the ledger. ThegetQueryResultForQueryStringWithPagination()
passes the queryString along with the pagesize and bookmark to the shim APIGetQueryResultWithPagination()
.
func (t *SimpleChaincode) queryMarblesWithPagination(stub shim.ChaincodeStubInterface, args []string) pb.Response {
// 0
// "queryString"
if len(args) < 3 {
return shim.Error("Incorrect number of arguments. Expecting 3")
}
queryString := args[0]
//return type of ParseInt is int64
pageSize, err := strconv.ParseInt(args[1], 10, 32)
if err != nil {
return shim.Error(err.Error())
}
bookmark := args[2]
queryResults, err := getQueryResultForQueryStringWithPagination(stub, queryString, int32(pageSize), bookmark)
if err != nil {
return shim.Error(err.Error())
}
return shim.Success(queryResults)
}
The following example is a peer command which calls queryMarblesWithPagination with a pageSize of 3
and no bookmark specified.
Tip
When no bookmark is specified, the query starts with the "first" page of records.
Try it yourself
// Rich Query with index name explicitly specified and a page size of 3:
peer chaincode query -C $CHANNEL_NAME -n marbles -c '{"Args":["queryMarblesWithPagination", "{\"selector\":{\"docType\":\"marble\",\"owner\":\"tom\"}, \"use_index\":[\"_design/indexOwnerDoc\", \"indexOwner\"]}","3",""]}'
The following response is received (carriage returns added for clarity), three of the five marbles are returned because the pagsize
was set to 3
:
[{"Key":"marble1", "Record":{"color":"blue","docType":"marble","name":"marble1","owner":"tom","size":35}},
{"Key":"marble2", "Record":{"color":"yellow","docType":"marble","name":"marble2","owner":"tom","size":35}},
{"Key":"marble3", "Record":{"color":"green","docType":"marble","name":"marble3","owner":"tom","size":20}}]
[{"ResponseMetadata":{"RecordsCount":"3",
"Bookmark":"g1AAAABLeJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqz5yYWJeWkGoOkOWDSOSANIFk2iCyIyVySn5uVBQAGEhRz"}}]
Note
Bookmarks are uniquely generated by CouchDB for each query and represent a placeholder in the result set. Pass the returned bookmark on the subsequent iteration of the query to retrieve the next set of results.
The following is a peer command to call queryMarblesWithPagination with a pageSize of 3
. Notice this time, the query includes the bookmark returned from the previous query.
Try it yourself
peer chaincode query -C $CHANNEL_NAME -n marbles -c '{"Args":["queryMarblesWithPagination", "{\"selector\":{\"docType\":\"marble\",\"owner\":\"tom\"}, \"use_index\":[\"_design/indexOwnerDoc\", \"indexOwner\"]}","3","g1AAAABLeJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqz5yYWJeWkGoOkOWDSOSANIFk2iCyIyVySn5uVBQAGEhRz"]}'
The following response is received (carriage returns added for clarity). The last two records are retrieved:
[{"Key":"marble4", "Record":{"color":"purple","docType":"marble","name":"marble4","owner":"tom","size":20}},
{"Key":"marble5", "Record":{"color":"blue","docType":"marble","name":"marble5","owner":"tom","size":40}}]
[{"ResponseMetadata":{"RecordsCount":"2",
"Bookmark":"g1AAAABLeJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqz5yYWJeWkmoKkOWDSOSANIFk2iCyIyVySn5uVBQAGYhR1"}}]
The final command is a peer command to call queryMarblesWithPagination with a pageSize of 3
and with the bookmark from the previous query.
Try it yourself
peer chaincode query -C $CHANNEL_NAME -n marbles -c '{"Args":["queryMarblesWithPagination", "{\"selector\":{\"docType\":\"marble\",\"owner\":\"tom\"}, \"use_index\":[\"_design/indexOwnerDoc\", \"indexOwner\"]}","3","g1AAAABLeJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqz5yYWJeWkmoKkOWDSOSANIFk2iCyIyVySn5uVBQAGYhR1"]}'
The following response is received (carriage returns added for clarity). No records are returned, indicating that all pages have been retrieved:
[]
[{"ResponseMetadata":{"RecordsCount":"0",
"Bookmark":"g1AAAABLeJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqz5yYWJeWkmoKkOWDSOSANIFk2iCyIyVySn5uVBQAGYhR1"}}]
For an example of how a client application can iterate over the result sets using pagination, search for the getQueryResultForQueryStringWithPagination
function in the Marbles sample.
It may be necessary to update an index over time. The same index may exist in subsequent versions of the chaincode that gets installed. In order for an index to be updated, the original index definition must have included the design document ddoc
attribute and an index name. To update an index definition, use the same index name but alter the index definition. Simply edit the index JSON file and add or remove fields from the index. Fabric only supports the index type JSON, changing the index type is not supported. The updated index definition gets redeployed to the peer’s state database when the chaincode is installed and instantiated. Changes to the index name or ddoc
attributes will result in a new index being created and the original index remains unchanged in CouchDB until it is removed.
Note
If the state database has a significant volume of data, it will take some time for the index to be re-built, during which time chaincode invokes that issue queries may fail or timeout.
If you have access to your peer's CouchDB state database in a development environment, you can iteratively test various indexes in support of your chaincode queries. Any changes to chaincode though would require redeployment. Use the CouchDB Fauxton interface or a command line curl utility to create and update indexes.
Note
The Fauxton interface is a web UI for the creation, update, and deployment of indexes to CouchDB. If you want to try out this interface, there is an example of the format of the Fauxton version of the index in Marbles sample. If you have deployed the BYFN network with CouchDB, the Fauxton interface can be loaded by opening a browser and navigating to http://localhost:5984/_utils
.
Alternatively, if you prefer not use the Fauxton UI, the following is an example of a curl command which can be used to create the index on the database mychannel_marbles
:
// Index for docType, owner. // Example curl command line to define index in the CouchDB channel_chaincode database
curl -i -X POST -H "Content-Type: application/json" -d
"{\"index\":{\"fields\":[\"docType\",\"owner\"]},
\"name\":\"indexOwner\",
\"ddoc\":\"indexOwnerDoc\",
\"type\":\"json\"}" http://hostname:port/mychannel_marbles/_index
Note
If you are using BYFN configured with CouchDB, replace hostname:port with localhost:5984
.
Index deletion is not managed by Fabric tooling. If you need to delete an index, manually issue a curl command against the database or delete it using the Fauxton interface.
The format of the curl command to delete an index would be:
curl -X DELETE http://localhost:5984/{database_name}/_index/{design_doc}/json/{index_name} -H "accept: */*" -H "Host: localhost:5984"
To delete the index used in this tutorial, the curl command would be:
curl -X DELETE http://localhost:5984/mychannel_marbles/_index/indexOwnerDoc/json/indexOwner -H "accept: */*" -H "Host: localhost:5984"