Home

Welcome to the HyperLedger Explained in a better style.

To start explaining what hyperledger fabric is and how it really works behind the hood, it's really hard for someone to know where to start explaining from. So, there might be the chance that I might explain things out of order.

HyperLedger fabric is a distributed ledger technology framework, which gives you the ability to build your own blockchain network in a very modular, flexible way and be in control all the time. Distributed ledger means that all the nodes that are part of the network have the copy of the same ledger and It's consensus what makes the things be in sync at any time. Hyperledger fabric offers so many advantages such as:

Permissioned network - If in bitcoin, anyone can join the network, mine the blocks and make transactions, here not anyone can join and only those can join whose identity is known and
Confidential transaction - This super feature allows us to have confidential transactions, meaning that I can make asset transaction to someone, which anyone in the network can see, but I can also make the transaction and I manage who can see it and who can't. This is achieved by private channels in fabric.
no cryptocurrency - if in bitcoin, you need cryptocurrency to be part of the network and make transactions, in fabric, only assets exist ( assets could be 'car', 'house', or whatever).
Programmable - it's super flexible meaning that you can choose block time, consensus algorithm and build your network as you wish.

Key concepts

Ledger

Every node in fabric has the copy of the ledger. Ledger is the recording book where all assets/transactions are kept. in fabric, Ledger also consists of two things - Transaction log and World-State Database.

Transaction Log - this is the place for a node where all transactions are kept and appended. It's immutable and written by using LevelDB database.
World-State Database - this is the place where the final results are kept. For example - if I had 10 cars, and I made the transaction 3 times and gave my friend 1 car each of the time, in Transaction log, there'll be 3 transactions appended, but in world-state, the old value(I had 10 cars), will be updated with (7 cars) and there'll be also version number appended to this updated value in world-state. Basically this version number appended is the key to solving double-spending.

MSP

Hyperledger fabric is framework that's organised for organizations. In order to join the network, you gotta be an organization. When added as organization, this organization adds its own peers so that it can get data and have its own copy of the ledger. Basically, what really happens is when registering as organization, you're given the certificate(certificate(public key+data) and private key), then when adding peers for this organization, peers also get the certificate(certificate(public key+data) and private key), but the key point is that peers' certificates are derived from its organization's certificate. This way peers are binded to organizations. MSP(Membership Service Provider) is a component that takes care of verifying which peers belong to which organizations and if peers' certificates are valid. If there's two organizations, orgA and orgB and orgA's peer does some operation on orgB, it won't be valid because orgA's peer is not part of orgB. Also, when having organizations and its own peers, there're also users who belong to specific organizations. Users have specific roles, so some of them can do some operations for their organization and some of them can do other staff, some of them can't do some operations.

In order for an organization to register a peer, first, organization needs to have certificate from which peers' certificates will be derived. Organizations have freedom to choose any certificate authority which will give this organization x.509 certificates. Then all the peers for this organization will have certificates derived from this organization's x.509 certificate.

Peer Types

In bitcoin or any other cryptocurrency blockchain, all peers are almost equal(the only difference is that some of them are full nodes and some of them light nodes). In fabric, peers can be different types and can do different kind of tasks. This way each peer knows what it needs to do and doesn't need to do everything. Pressure and doing all the tasks by peer is reduced this way.

Peer types:

Orderer: Orderer gets the transaction endorsed response, puts it in a new block and shares it to organizations' leader peers.
Peer:
- Commiter Peers - All the peers in fabric by default are Commiter peers. Commiter peer's only responsibility is to get blocks from other peers and update their ledger.
- Endorsing Peers- This kind of peer takes care of executing transactions and getting results. In order for a peer to be Endorsing peer, it needs to have chaincode installed on it.
- Anchor peers - This kind of peers are special type peers and they're known outside the organization. Other peers(not anchor) are only known inside the organization. So if peer is not anchor, then orgA's peer can't discover orgB's peer. If orgA and orgB have anchor peers in it, They use gossip protocol to talk to each other. (why needed ? later will be explained).
- Leader peers - This peer is the only one in each organization whose responsibility is to get blocks from orderers and share it with other peers inside the organization. Leader peers can be elected by automatically by all peers in the organization.
Client: This is just an application which uses node sdk to contact with fabric network.

Peer types above are described, but transaction flow will better help us to find out what each peer does and why.

Transaction Flow

Let's say we have two organizations(orgA and orgB) and in each organization, we have 1 endorsing peer, 1 anchor peer, 1 commiter peer. We also have another organization(orgC) who has the orderer peer.

Let's say userA from node sdk wants to make transaction and transfer '10 cars' to someone else. This userA is part of orgA. When this userA makes the transaction proposal, it should specify that this request must go to orgA's endorsing peer(because when transaction is update-type, only chaincode can invoke transaction on a ledger, so as we said, endorsing peers are the one who has the chaincode installed on it). When UserA's proposal goes to orgA's endorsing peer, endorsing peer first checks that this proposal is signed by the valid certificate. Then it invokes chaincode transfer function, chaincode transfer function returns the result which would be if this proposal could be put into ledger. So what endorsing peer did is it simulated this transaction(didn't update the ledger), then put this result into Read-Write Set, then peer signed on this and returned final response(endorsment response) to userA's sdk. userA then takes this response and sends it to orderer node.Orderer does not need to inspect the entire content of a transaction in order to perform its operation, it simply receives transactions from all channels in the network, orders them chronologically by channel, and creates blocks of transactions per channel. Now orderer will share this block to organizations' leader peers. Leader peers will also share this block to other peers in the organization(commiter, endorsing(if it exists), anchor). Those peers will check this block's transactions(if they're valid, if endorsement policy, and if RW set are correct). Then they'll update their ledger, it doesn't matter if transaction is valid or invalid, they will still put any kind of transaction into their ledger, but flag it with 'valid' or 'invalid'.

Sometimes, when an admin installs chaincode on a peer, it also specifies endorsment policy. if we have orgA and orgB and we installed chaincode on orgA's peer and orgB's peer, when instantiated chaincode, we could say something like this: whenever someone executes this chaincode's function, it must be executed by more than 1 peer. We might need the following that when userA made the request to orgA's peer, it also wants to get the response for the same request from orgB's peer. this way if our request is done by two peers from different organizations, It's more likely that the answer is more solid. I won't go details in that. So when instantated chaincode on orgA's peer, we could say that whoever passes transaction request to this chaincode, the response won't be valid unless orgB's peer also executes this chaincode. so what UserA has to do is if he knows that this chaincode has endorsement policy like this, he should send his transaction proposal to both (orgA's peer and orgB's peer). User will wait for two responses, then he will send these two responses to orderer, orderer will give it to leaders. what leaders, commiters and anchors will check is if RW are correct and also they'll know that this transaction must be calculated by two peers and these two peers' responses must be the same. If not the same or only 1 peer signed it, then transaction will be flagged as invalid. The good question might be that if commiters, anchors and leaders aren't required to have chaincode, when they get transactions from orderer, how do they check the endorsement policy if they don't have chaincode? Answer is the following: the endorsement policy is distributed to all peers in a channel when chaincode is instantiated even if the peer does not have the chaincode bytes. IBM developer Gari explained it here : https://stackoverflow.com/questions/50461594/chaincode-should-only-be-installed-on-endorsing-peer-nodes?rq=1&fbclid=IwAR3a2mBN863dc_TSRHKOWFzToa3ffnS6RJPHT-UNQn70daa1QKmPbO7Dp6g. Also if you understand all of this, you might ask that if I want endorsement policy to be executed from all the organizations(if having 10 organizations, and only 9 will execute it, it gets invalid), it's tiresome for user to know all the endorsing peers' addresses to send this transaction proposal. So fabric made it easier for us by using Discovery Service. You can find it more here: https://hyperledger-fabric.readthedocs.io/en/release-1.3/discovery-overview.html.

How gossip works?

First thing first. Why do we need gossip protocol? It will be more helpful if we imagine scenarios and explain it this way.

A) - Let's say we have orgA and there're many peers in this orgA. In reality, inside the organization, peers know each other. They're like a connected graph.

Gossip 1: So when leader gets the block from orderer, what should happen is it has to share this block to other peers.
Gossip 2: If in orgA commiter peer somehow has the ledger in which there're 10 blocks,and leader has 14 blocks, This might happen because leader might not be able to share it with commiter peers(due to error or couldn't discover it). What really happens is even if commiter knows about leader peer, it will gossip and get the remaining 4 blocks from leader peer.
Gossip 3: If we add new peer(commiter for example) in orgA, if it doesn't have the ability to gossip, it would never be able to get the blocks that other peers already had before adding this new peer. So this new peer will gossip and get those blocks.
Gossip 4: What if commiter peer shuts down,and when it gets shut down, others in orgA still keep going and they have already 40 blocks, but then commiter got shut down, they had 30, so what Admin will do is update/relaunch the commiter peer,and then commiter will gossip with others and get those 10 blocks.

So gossip helps us solve this problem. I only explained the situation within the organization. In order for an organizations' peers to see each other, we must use CORE_PEER_GOSSIP_BOOTSTRAP for each peer.

Scenario 1: This means that if we set this config for peer0 and value will be peer1, peer2,peer3. then peer0 will be able to discover peer1,peer2,peer3. This way, if peer3 is connected to peer4, but peer0 is not connected to peer4, peer0 will still discover peer4, because it first finds peer3 and then finds peer4.
Scenario 2: If peer0 knows about peer4 and peer4 doesn't know about peer0, when peer0 starts communicating with peer4, then peer4 also knows about peer0.

B) - So let's say we have two org(orgA, orgB). What can do we to make orgA's peer discoverable from orgB's peer. The first option is to set orgA's peer and orgB's peer anchor peers, which means they'll discover each other even though they're in different organizations. This config (who is anchor peer) is put into genesis block. By using anchor peers, here is how it can help us.

Gossip 1: if orgA's anchor peer has 10 blocks, orgB's anchor peer has 15 blocks, they will gossip and orgA's anchor peer will update its ledger(will add 5 blocks to its 10). I also forgot to mention that anchor peer also can share the block with other peers in the same organization(don't think that only leader peer can do that). So anchor peers gossip through organizations to get synched.
Gossip 2: what if we add new organization in the same channel and when it's added, how is this new organization's peer are going to get already added blocks in other organizations? what happens is this new organization's anchor peer will gossip with others and get the blocks, then this anchor will share the blocks with peers in the same organization.
Gossip 3: ,same thing as mentioned above - what if peers die in the organization? when we update peers(fix them), they have to get lost data again.

Also from different organizations, not only anchor peers can discover each other. We can set it by hand so that to give power to orgA's commiter peer to discover orgB's commiter peer. How? Basically, Nice answer can be followed on this stackoveflow link where Gari described two properties : https://stackoverflow.com/questions/51377474/hyperledger-fabric-gossip-bootstrap-gossip-externalendpoints

Basically, In the following section, I'll explain where genesis block is, where configurations are stored, what kind of code we have to write in order to run the fabric network and also use private channel for confidential transactions.

The following example will have 4 organizations. 1 of them is organization that has orderer node running. Let's say each of other 3 organizations has only 1 peer and let's make this 1 peer in each organization anchor peer.

This demo will only explain how to run in development mode. Wherever you see cryptogen tool, it's not good to use it in production. Production we use fabric-ca.

The first step is to have certificates for organization and its own peers.

OrdererOrgs:
  - Name: Orderer
    Domain: example.com
    Specs:
      - Hostname: orderer

PeerOrgs:
  - Name: Org1
    Domain: org1.example.com
    Template:
      Count: 1
    Users:
      Count: 1

  - Name: Org2
    Domain: org2.example.com
    Template:
      Count: 1
    Users:
      Count: 1
  
  - Name: Org3
    Domain: org3.example.com
    Template:
      Count: 1
    Users:
      Count: 1

Name is nothing else but just the name for a developer to extinguish. Domain will be the folder name where certificates will be put. Template means each organization will have only 1 peer and its names(folders) will be peer0.org1.example.com, peer0.org2.example.com, peer0.org3.example.com. Users - we have 1. It means one user will be created and also Admin user by default.

./cryptogen generate --config=./cryptogen.yaml --output=../materials/crypto-config This gives us folders, crypto-materials in it.

Now the second step is to bind organizations certificates and peers certificates each other. Let's move on to configtx.yaml file.

---

Organizations:
    #organizations section tells us the following. We give some id to organizations itself and also specify
    #organisation's crypto materials itself. When we run the peers in the following section, we will specify the 
    #id of the organization and crypto materials for the peer so that it will be binded.
    - &OrdererOrg
        Name: OrdererMSP
        ID: OrdererMSP
        MSPDir: ../materials/crypto-config/ordererOrganizations/example.com/msp

    - &Org1
        Name: Org1MSP
        ID: Org1MSP
        MSPDir: ../materials/crypto-config/peerOrganizations/org1.example.com/msp
        AnchorPeers:
            - Host: peer0.org1.example.com
              Port: 7051

    - &Org2
        Name: Org2MSP
        ID: Org2MSP
        MSPDir: ../materials/crypto-config/peerOrganizations/org2.example.com/msp
        AnchorPeers:
            - Host: peer0.org2.example.com
              Port: 7051
    
    - &Org3
        Name: Org3MSP
        ID: Org3MSP
        MSPDir: ../materials/crypto-config/peerOrganizations/org3.example.com/msp
        AnchorPeers:
            - Host: peer0.org3.example.com
              Port: 7051


Application: &ApplicationDefaults
    Organizations:

# This is easy. BatchTimeout and BatchSize means how much time it should take to create blocks,
# how much time to wait before sending the blocks to leader.
Orderer: &OrdererDefaults
    
    OrdererType: solo
    Addresses:
        - orderer.example.com:7050

    BatchTimeout: 2s
    BatchSize:
        MaxMessageCount: 10
        AbsoluteMaxBytes: 98 MB
        PreferredMaxBytes: 512 KB

    Organizations:


Profiles:

    ThreeOrgGenesisBlock:
        Orderer:
            <<: *OrdererDefaults
            Organizations:
                - *OrdererOrg
        Consortiums:
            SampleConsortium:
                Organizations:
                    - *Org1
                    - *Org2
                    - *Org3
    ThreeOrgsChannel:
        Consortium: SampleConsortium
        Application:
            <<: *ApplicationDefaults
            Organizations:
                - *Org1
                - *Org2
                - *Org3

Here, I'll talk about these profiles section in configtx.yaml, because it's the most important one. Let's look at ThreeOrgGenesisBlock section. First of all, in order for a network to work, before running the network, there must be a genesis block. This genesis block is used by orderer node. what we put here is that network should consist of 3 organizations and *Org1 this means that above we specifed Org1's MSP configuration. * means that it will take above Org1's settings and put it here. So in genesis block, there'll be organizations that should be part of the network, their msp configurations and orderer node's settings. After that what we do is we generate the genesis block according to this setting, I just talked about.

./configtxgen -profile ThreeOrgGenesisBlock -outputBlock ../materials/genesis.block

When running the network, this genesis block will be passed to orderer's node. What will happen at this stage is the key point. When we run the network, there'll be 3 organizations part of it(these 3 organizations still won't be part of any channel, even the global channel), but for an orderer node, there'll be new channel created - it's bootstrapped by default and this channel's name is orderer system channel, orderer will be joined to this channel and only it will be part of it. in this channel, orderer will have the ledger and in this ledger, there'll be this genesis block appended.

Now, we have to create the global channel which organizations and its peers will be part of. The code for it is in the profiles section called: 'ThreeOrgsChannel'. What we have said here is that in this channel, Org1, Org2, Org3 will be part of. First, we have to generate the file where this configuration will be saved.

./configtxgen -profile ThreeOrgsChannel -outputCreateChannelTx ../materials/channel.tx -channelID "GlobalChannel".

This command gave us the file called "channel.tx".

Now we can run the network. We have genesis block and we have channel.tx files. To run a network, we need configuration for peers and if we use certificate authority for each organisation, we have to have the service running for these certificate authorities.

#
# Copyright IBM Corp. All Rights Reserved.
#
# SPDX-License-Identifier: Apache-2.0
#
version: '2'

services:

  fabric_ca_for_org1:
    image: hyperledger/fabric-ca
    environment:
      - FABRIC_CA_HOME=/etc/hyperledger/fabric-ca-server
      - FABRIC_CA_SERVER_CA_NAME=fabric_ca_for_org1
      - FABRIC_CA_SERVER_CA_CERTFILE=/etc/hyperledger/fabric-ca-server-config/ca.org1.example.com-cert.pem
      - FABRIC_CA_SERVER_CA_KEYFILE=/etc/hyperledger/fabric-ca-server-config/379210cbaff9b700914148154bb77444baedda02ecca9fcbacef42374c227f87_sk
      - FABRIC_CA_SERVER_TLS_ENABLED=true
      - FABRIC_CA_SERVER_TLS_CERTFILE=/etc/hyperledger/fabric-ca-server-config/ca.org1.example.com-cert.pem
      - FABRIC_CA_SERVER_TLS_KEYFILE=/etc/hyperledger/fabric-ca-server-config/379210cbaff9b700914148154bb77444baedda02ecca9fcbacef42374c227f87_sk
      - GODEBUG=netdns=go
    ports:
      - "7054:7054"
    command: sh -c 'fabric-ca-server start -b admin:adminpw -d'
    volumes:
      - ./materials/crypto-config/peerOrganizations/org1.example.com/ca/:/etc/hyperledger/fabric-ca-server-config
    container_name: fabric_ca_for_org1

  fabric_ca_for_org2:
    image: hyperledger/fabric-ca
    environment:
      - FABRIC_CA_HOME=/etc/hyperledger/fabric-ca-server
      - FABRIC_CA_SERVER_CA_NAME=fabric_ca_for_org2
      - FABRIC_CA_SERVER_CA_CERTFILE=/etc/hyperledger/fabric-ca-server-config/ca.org2.example.com-cert.pem
      - FABRIC_CA_SERVER_CA_KEYFILE=/etc/hyperledger/fabric-ca-server-config/780ad76518f41bbd0947788c705080d0abe346e78d26c7d2dc0b1d95e76e1e0e_sk
      - FABRIC_CA_SERVER_TLS_ENABLED=true
      - FABRIC_CA_SERVER_TLS_CERTFILE=/etc/hyperledger/fabric-ca-server-config/ca.org2.example.com-cert.pem
      - FABRIC_CA_SERVER_TLS_KEYFILE=/etc/hyperledger/fabric-ca-server-config/780ad76518f41bbd0947788c705080d0abe346e78d26c7d2dc0b1d95e76e1e0e_sk
      - GODEBUG=netdns=go
    ports:
      - "8054:7054"
    command: sh -c 'fabric-ca-server start -b admin:adminpw -d'
    volumes:
      - ./materials/crypto-config/peerOrganizations/org2.example.com/ca/:/etc/hyperledger/fabric-ca-server-config
    container_name: fabric_ca_for_org2

  
  fabric_ca_for_org3:
    image: hyperledger/fabric-ca
    environment:
      - FABRIC_CA_HOME=/etc/hyperledger/fabric-ca-server
      - FABRIC_CA_SERVER_CA_NAME=fabric_ca_for_org3
      - FABRIC_CA_SERVER_CA_CERTFILE=/etc/hyperledger/fabric-ca-server-config/ca.org3.example.com-cert.pem
      - FABRIC_CA_SERVER_CA_KEYFILE=/etc/hyperledger/fabric-ca-server-config/b18f7344158c4a5078ffb92da826cd4d549ac15f522e6580a44d1bbe53ad8918_sk
      - FABRIC_CA_SERVER_TLS_ENABLED=true
      - FABRIC_CA_SERVER_TLS_CERTFILE=/etc/hyperledger/fabric-ca-server-config/ca.org3.example.com-cert.pem
      - FABRIC_CA_SERVER_TLS_KEYFILE=/etc/hyperledger/fabric-ca-server-config/b18f7344158c4a5078ffb92da826cd4d549ac15f522e6580a44d1bbe53ad8918_sk
      - GODEBUG=netdns=go
    ports:
      - "9054:7054"
    command: sh -c 'fabric-ca-server start -b admin:adminpw -d'
    volumes:
      - ./materials/crypto-config/peerOrganizations/org3.example.com/ca/:/etc/hyperledger/fabric-ca-server-config
    container_name: fabric_ca_for_org3





  orderer.example.com:
    container_name: orderer.example.com
    image: hyperledger/fabric-orderer
    environment:
      - ORDERER_GENERAL_LOGLEVEL=debug
      - ORDERER_GENERAL_LISTENADDRESS=0.0.0.0
      - ORDERER_GENERAL_GENESISMETHOD=file
      - ORDERER_GENERAL_GENESISFILE=/etc/hyperledger/configtx/genesis.block
      - ORDERER_GENERAL_LOCALMSPID=OrdererMSP
      - ORDERER_GENERAL_LOCALMSPDIR=/etc/hyperledger/crypto/orderer/msp
      # - ORDERER_GENERAL_TLS_ENABLED=true
      # - ORDERER_GENERAL_TLS_PRIVATEKEY=/etc/hyperledger/crypto/orderer/tls/server.key
      # - ORDERER_GENERAL_TLS_CERTIFICATE=/etc/hyperledger/crypto/orderer/tls/server.crt
      # - ORDERER_GENERAL_TLS_ROOTCAS=[/etc/hyperledger/crypto/orderer/tls/ca.crt, /etc/hyperledger/crypto/peerOrg1/tls/ca.crt, /etc/hyperledger/crypto/peerOrg2/tls/ca.crt]
      - GODEBUG=netdns=go
    working_dir: /opt/gopath/src/github.com/hyperledger/fabric/orderers
    command: orderer
    ports:
      - 7050:7050
      ## why do we pass here peerOrganizations artifacts?
    volumes:
        - ./materials:/etc/hyperledger/configtx
        - ./materials/crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/:/etc/hyperledger/crypto/orderer
        - ./materials/crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/:/etc/hyperledger/crypto/peerOrg1
        - ./materials/crypto-config/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/:/etc/hyperledger/crypto/peerOrg2
        - ./materials/crypto-config/peerOrganizations/org3.example.com/peers/peer0.org3.example.com/:/etc/hyperledger/crypto/peerOrg3

  peer0.org1.example.com:
    container_name: peer0.org1.example.com
    extends:
      file:   base.yaml
      service: peer-base
    environment:
      - GOPATH=/opt/gopath
      - CORE_PEER_ID=peer0.org1.example.com
      - CORE_PEER_LOCALMSPID=Org1MSP
      - CORE_PEER_ADDRESS=peer0.org1.example.com:7051
      - CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org1.example.com:7051      
      - CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.org1.example.com:7051
      - CORE_CHAINCODE_KEEPALIVE=10
    ports:
      - 7051:7051
      - 7053:7053
    volumes:
        - ./src/github.com/:/opt/gopath/src/github.com/
        - ./materials:/etc/hyperledger/configtx
        - ./materials/crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/:/etc/hyperledger/peer
        - ./materials/crypto-config/peerOrganizations/org1.example.com/users/:/etc/hyperledger/users
    depends_on:
      - orderer.example.com

  

  peer0.org2.example.com:
    container_name: peer0.org2.example.com
    extends:
      file:   base.yaml
      service: peer-base
    environment:
      - CORE_PEER_ID=peer0.org2.example.com
      - CORE_PEER_LOCALMSPID=Org2MSP
      - CORE_PEER_ADDRESS=peer0.org2.example.com:7051
      - CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org2.example.com:7051      
      - CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.org2.example.com:7051
      - GODEBUG=netdns=go
    ports:
      - 8051:7051
      - 8053:7053
    volumes:
        - ./materials:/etc/hyperledger/configtx
        - ./materials/crypto-config/peerOrganizations/org2.example.com/peers/peer0.org2.example.com/:/etc/hyperledger/peer
        - ./materials/crypto-config/peerOrganizations/org2.example.com/users/:/etc/hyperledger/users
    depends_on:
      - orderer.example.com

  
  peer0.org3.example.com:
    container_name: peer0.org3.example.com
    extends:
      file:   base.yaml
      service: peer-base
    environment:
      - CORE_PEER_ID=peer0.org3.example.com
      - CORE_PEER_LOCALMSPID=Org3MSP
      - CORE_PEER_ADDRESS=peer0.org3.example.com:7051
      - CORE_PEER_GOSSIP_BOOTSTRAP=peer0.org3.example.com:7051      
      - CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.org3.example.com:7051
      - GODEBUG=netdns=go
    ports:
      - 9051:7051
      - 9053:7053
    volumes:
        - ./materials:/etc/hyperledger/configtx
        - ./materials/crypto-config/peerOrganizations/org3.example.com/peers/peer0.org3.example.com/:/etc/hyperledger/peer
        - ./materials/crypto-config/peerOrganizations/org3.example.com/users/:/etc/hyperledger/users
    depends_on:
      - orderer.example.com

So we write docker-compose up and containers starts running. network is ready. as you can see, in orderer's configuration above, we passed genesis block. Now network is running, but we need this organizations' peers to have the global channel and join that channel. They aren't still part of any channel yet. but we have channel.tx file which will help us create channel and join it.

docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/users/Admin@org1.example.com/msp" peer0.org1.example.com peer channel create -o orderer.example.com:7050 -c GlobalChannel -f /etc/hyperledger/configtx/channel.tx

docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/users/Admin@org1.example.com/msp" peer0.org1.example.com peer channel join -b GlobalChannel.block

What we did is use org1's peer0's container and created channel transaction by using channel.tx file. When creating channel transaction, what happens is it goes to orderer(it's kind of a transaction). what orderer does is it also gets part of new channel called "GlobalChannel"(the one I named my channel, it was part of orderer system channel first). So it's part of both now. In GlobalChannel's ledger, it puts this configuration(channel.tx) as genesis block. so when executed the first command, channel got created and it gave us GlobalChannel.block file. Now this file is used to join the network(only admin could instantiate these commands). When peer joins this channel, it gets part of this channel, has new ledger, and this configuration setting as genesis block in that channel's ledger.

Key part 1: We derived this channel.tx file from profiles section where we put 3 of them as part of this channel. If we could only put two organizations in that profiles channel settings, what would happen is org3 wouldn't be able to join this channel, because this channel.tx got configured only for org1 and org2.
Key part 2: after org1's admin got the channel.block and joined the channel, he should send this channel.block to those organizations who has the right to join this channel. Those organizations' developers will take care of their peers joining this channel.
Key part 3: each peer has to join the channel separately.

Now what we can do is install the chaincode on these peers and invoke some functions.

docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/users/Admin@org1.example.com/msp" peer0.org1.example.com peer chaincode install -n fabcar -v 1.0 -p /opt/gopath/src/github.com/fabcar/node -l "NODE"
docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/users/Admin@org1.example.com/msp" peer0.org1.example.com peer chaincode instantiate -o orderer.example.com:7050 -C giochannel -n fabcar -l "NODE" -v 1.0 -c '{"Args":[""]}'
sleep 10
docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_MSPCONFIGPATH=/etc/hyperledger/users/Admin@org1.example.com/msp" cli peer chaincode invoke -o orderer.example.com:7050 -C giochannel -n fabcar -c '{"function":"initLedger","Args":[""]}'

Peer0 from org1 installs the chaincode (which only means that some specific binary file gets installed on org1's peer0's hard disk). Then instantiating means that we're making a specific kind of transaction which will make this chaincode on peer0 available for working in network. So peer0 in org1 has the chaincode installed. You can see how invoking query works for example right away. If we want to install this chaincode again(maybe we changed some part), we update the version and do it again. Other peers from other organisations have to install it with the same name.

Some Key Parts we might be missing.

1) What does orderer node do ?

The configuration of a channel may be updated using a channel configuration update transaction. This transaction contains a representation of the changes to be made to the configuration, as well as a set of signatures. The ordering service nodes evaluate whether the update is valid by using the current configuration to verify that the modifications are authorized using the signatures. The order- ers then generate a new configuration block, which embeds the new configuration and the configuration update transaction. Peers receiving this block validate whether the configuration update is authorized based on the current configuration; if valid, they update their current configuration.
Getting transactions, making them in order and broadcasting it to leaders.
Before broadcasting, it also checks the policy for the chaincode (if clientA invokes some function on a chaincode and gives the endorsment responses to orderer, it checks that the submitter has the appropriate(for-example write) access to the channel or not. If client doesn't have the permission, orderer even restricts receiving blocks to peers and broadcasting transactions. This is different from endorsment policy. Orderer doesn't check endorsement policy at all.

2) how does peer check the transaction is valid or not that it got from orderer via block? https://stackoverflow.com/questions/53581922/what-will-happen-to-block-if-more-than-one-transaction-are-changing-same-asset-i/53583575#53583575

only organization's admin users can create channels.(enrolled users can't do that, peer admins can't do that).
only peer admins can join the channel(organization's admins can't do that, enrolled users can't do that).
only peer admins can install the chaincode(organization's admins can't do that, enrolled users can't do that).
only organization admins can instantiate the chaincode(peer admins can't do that, enrolled users can't do that).
You can still add organization to a channel even if the organization is not in orderer's system channel's genesis block (I mean in the consortium). But you can't put an organization into a channel creation's profile section if this organization is not put into a consortium in orderer's genesis block profile section(but this restriction only applies when you first start the network).
ACL stackoverflow - https://stackoverflow.com/questions/51670810/implementing-acl-in-hyperledger-fabric-v1-2
There might be 4 type of roles.

Admin - wherever we see admin role, it's required to have organization's admin's certificate.
Member - anyone that its parent organization issued certificate for.
Client and Peer could also be. THere's a notion of NodeOus which puts specific keyword in a certificate so that we could notify if it's peer or client or whoever.

Nice helpful link about policies and trees https://hyperledger-fabric.readthedocs.io/en/release-1.3/policies.html

Understanding Debugger Logs

I also looked through container's debugging mode and I'll post what kind of things happen for different kind of operations.

Create Channel Transaction.

This operation is an administrative task(couldn't be controlled by ACL).

My channel name is globalchannel.

orderer - Processing channel create tx for channel globalchannel on system channel testchainid
orderer goes to its system-channel's genesis block, takes all organization that are in this channel.tx, and checks if organization's certificate and it's admin certificate are connected (derived) and valid.
orderer starts evaluating policy /Channel/Application/ChannelCreationPolicy(this implicit meta policy by default has Any Admins) which takes us to /Channel/Application/ShopOrgMSP/admins. Then here it checks if signature which came is really the admin of ShopOrgMSP.
if signature is type of admin from (3), then it checks channel/writers(any writers) implicit meta policy. which takes us to channel/orderer/writers(any writers) which also takes us to channel/orderer/ordererorg/writers which has the signature type with a rule : ordererorg.member. then checks signatures if all good, continues.
then creates the channel which has all the default channel configurations and policies except the ones we added in Applications section

Join channel

can't be controlled by ACL and is controlled BY CSCC when joining a channel, peer first needs to get genesis block from orderer.

when we do getGenesisBlock, we specify the peer's admin's certificates. here is what happens: First policy that gets executed is /Channel/Readers(any readers) which execute its two children policy(/Channel/Application/Readers and /Channel/Orderer/Readers). if one of them returns true, we are good to go. When we specifed peer's admin certificate, it first checks /Channel/Orderer/OrdererOrg/Readers and its signature must be from OrdererMSP organization, we had ShopOrg's peers certificate.so this policy returns false. Then we move on to /Channel/APplication/ShopOrg/Readers(member of ShopOrg signature) and signature gets satisfied. So reading genesis block works. Now peer has to put this block into its new channel. IF we don't specify peer's admin certificate, after reading the block, it won't be able to put this block(permission denied). so there's a Configuration System Chaincode(CSCC) which handles channel configuration on the peer and in it, there's a strict rule that only peers Admins are allowed to join the channel.

Install Chaincode

(can't be controlled by ACL and is controlled By LSCC)

when installing chaincode, it only installs chaincode on a peer, but it's not known to channel and nothing can be done by this operation only. only peers admins are allowed to do this operation. The policy that takes care of verifying if it's peer admin or not is via LSCC. (lifecycle system chaincode)

Querying chaincode

Is controlled By ACL (peer/Propose)

executes peer/Propose ACL which has /Channel/Application/Writers by default. It depends on what we have in /Channel/Application/Writers. by default, it has ANY WRITERS. to better understand we could do something like this:

peer/Propose: /Channel/Application/MyPolicy1
MyPolicy1:
    Type: Signature
    Rule: "OR('ShopOrgMSP.admin')"

So now only ShopOrgMSP admin can query the chaincode and no one else at all. even the member of ShopOrgMSP.

Invoking chaincode