README

---

PyEthFinality

• Trello Board (Project Planning) - https://trello.com/b/rkTmcLNw/pyethfinality

Table of Contents

• Chapter 0 - Quick Start Guide
• Chapter 1 - Docker Setup
• Chapter 2 - Tests
• Chapter 3 - Editor / IDE
• Chapter 100 - Contributors, Bugs & Issues
• Chapter 999 - Unsorted
• Chapter 1000 - ERRORS - macOS Unix Bash Setup Attempt

Chapter 0 - Quick Start Guide

• Clone https://github.com/ltfschoen/pyethfinality

• Install and run Docker for Mac

• Run docker-compose up --force-recreate --build -d

• Get the container_id with docker ps -l

• Create 1st Bash Terminal tab and run interactive Docker shell (for running Python script) with docker exec -it <container_id> bash

• Create 2nd Bash Terminal tab and run interactive Docker shell (for TestRPC) with docker exec -it <container_id> bash

• Create 3rd Bash Terminal tab and run interactive Docker shell (for cURL requests to API) with docker exec -it <container_id> bash

• Start up the TestRPC in the 2nd Bash Terminal's Docker shell

  rm -rf ./db;
  mkdir -p db/chaindb;
  testrpc --account '0x0000000000000000000000000000000000000000000000000000000000000001, 50002471238800000000000' \
    --account '0x0000000000000000000000000000000000000000000000000000000000000002, 370004471238800000000000' \
    --unlock '0x0000000000000000000000000000000000000000000000000000000000000001' \
    --unlock '0x0000000000000000000000000000000000000000000000000000000000000002' \
    --blocktime 0 \
    --deterministic true \
    --port 8545 \
    --hostname localhost \
    --gasPrice 20000000000 \
    --gasLimit 0x47E7C4 \
    --debug true \
    --mem true \
    --db './db/chaindb'
  

• Run the Python script in the 1st Bash Terminal tab's Docker shell

  python3 ./scripts/src/main.py
  

• Run cURL request to simulate request to API from a front-end Client App in the 3rd Bash Terminal tab's Docker shell

  curl -i 127.0.0.1:5000/api/v1.0/query?query=None&median_balance=True
  

  • Response containing median account balance

    root@ed71f34ac199:/code# curl -i 127.0.0.1:5000/api/v1.0/query?query=None&median_balance=True
    [1] 204
    root@ed71f34ac199:/code# HTTP/1.0 200 OK
    Content-Type: application/json
    ContentType: application/json
    Content-Length: 4264
    Server: Werkzeug/0.12.2 Python/3.6.3
    Date: Fri, 01 Dec 2017 20:42:28 GMT
    
    {
      "longest_chain": {
        "block_count": 3, 
        "blocks": [
          {
            "accounts": [
              {
                "_address": "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf", 
                "_balance": 1000
              }, 
              {
                "_address": "0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF", 
                "_balance": 0
              }
            ], 
            "block_hash": "0xfb00000000000000000000000000000000000000000000000000000000000000", 
            "created_at": "2017-12-01 20:42:14.61162", 
            "id": [
              1
            ], 
            "parent_block_hash": "0x2d3816b0c954bf9495cf1eae723f73ae8c9f140622a07f5ba69f0897a8bae72a", 
            "transactions": [
              {
                "account_from": "0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF", 
                "account_id": "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf", 
                "account_to": 5000, 
                "amount": "pending", 
                "probability_reverted": "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf", 
                "status": 0.5, 
                "tx_receipt": null
              }, 
              {
                "account_from": "0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF", 
                "account_id": "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf", 
                "account_to": 34000, 
                "amount": "pending", 
                "probability_reverted": "0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF", 
                "status": 0.5, 
                "tx_receipt": null
              }
            ]
          }, 
          {
            "accounts": [
              {
                "_address": "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf", 
                "_balance": 1000
              }, 
              {
                "_address": "0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF", 
                "_balance": 0
              }
            ], 
            "block_hash": "0x5a00000000000000000000000000000000000000000000000000000000000000", 
            "created_at": "2017-12-01 20:42:14.66245", 
            "id": [
              2
            ], 
            "parent_block_hash": "0xfb00000000000000000000000000000000000000000000000000000000000000", 
            "transactions": [
              {
                "account_from": "0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF", 
                "account_id": "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf", 
                "account_to": 5000, 
                "amount": "pending", 
                "probability_reverted": "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf", 
                "status": 0.5, 
                "tx_receipt": null
              }, 
              {
                "account_from": "0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF", 
                "account_id": "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf", 
                "account_to": 34000, 
                "amount": "pending", 
                "probability_reverted": "0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF", 
                "status": 0.5, 
                "tx_receipt": null
              }
            ]
          }, 
          {
            "accounts": [
              {
                "_address": "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf", 
                "_balance": 1000
              }, 
              {
                "_address": "0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF", 
                "_balance": 0
              }
            ], 
            "block_hash": "0x5100000000000000000000000000000000000000000000000000000000000000", 
            "created_at": "2017-12-01 20:42:14.75240", 
            "id": [
              3
            ], 
            "parent_block_hash": "0x5a00000000000000000000000000000000000000000000000000000000000000", 
            "transactions": [
              {
                "account_from": "0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF", 
                "account_id": "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf", 
                "account_to": 5000, 
                "amount": "pending", 
                "probability_reverted": "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf", 
                "status": 0.5, 
                "tx_receipt": null
              }, 
              {
                "account_from": "0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF", 
                "account_id": "0x7E5F4552091A69125d5DfCb7b8C2659029395Bdf", 
                "account_to": 34000, 
                "amount": "pending", 
                "probability_reverted": "0x2B5AD5c4795c026514f8317c7a215E218DcCD6cF", 
                "status": 0.5, 
                "tx_receipt": null
              }
            ]
          }
        ], 
        "median_account_balance": 1000
      }
    }
    

• Run PyTest in Docker shell

  python3 -m pytest .
  

Chapter 1 - Docker Setup

• Docker

  • Run Docker for Mac

  • Run Docker Container / Remove existing Docker network from image. Use Ubuntu packages listed at: https://packages.ubuntu.com/search?suite=all&section=all&arch=any&keywords=solc&searchon=names

    docker-compose down
    docker-compose up --force-recreate --build -d
    

  • Open Docker sandbox session (Multiple tabs)

    docker ps -l
    
    # run interactive shell to the container 
    docker exec -it <container_id> bash
    
    # create other interaction shells to the container (to run TestRPC)
    docker exec -it <container_id> bash
    

  • Open Docker sandbox session (Single tab)

    docker-compose exec sandbox bash
    

    • Show Linux version - http://docs.python-guide.org/en/latest/starting/install3/linux/

      cat /proc/version
      python3 --version
      solc --help
      
      export LC_ALL=C.UTF-8
      export LANG=C.UTF-8
      populus
      

      • TODO - Add PyEnv and switch to latest version of Python 3

  • Run (Docker terminal tab 2)

    rm -rf ./db;
    mkdir -p db/chaindb;
    testrpc --account '0x0000000000000000000000000000000000000000000000000000000000000001, 10002471238800000000000' \
      --account '0x0000000000000000000000000000000000000000000000000000000000000002, 10004471238800000000000' \
      --unlock '0x0000000000000000000000000000000000000000000000000000000000000001' \
      --unlock '0x0000000000000000000000000000000000000000000000000000000000000002' \
      --blocktime 0 \
      --deterministic true \
      --port 8545 \
      --hostname localhost \
      --gasPrice 20000000000 \
      --gasLimit 0x47E7C4 \
      --debug true \
      --mem true \
      --db './db/chaindb'
    

  • Run (Docker terminal tab 1)

    python3 ./scripts/src/main.py
    

  • Other

    • Show where Python Packages are installed

      python3 -m site
      which /root/.py-solc/solc-v0.4.17/bin/solc
      python3 -m solc.install v0.4.17
      

    • Note: Any files created or modified in the /code directory of the Docker VM are synchronised with this project
    • Note: Manually compile Solidity smart contract with solc --bin -o $PWD/solcoutput dapp-bin=/usr/local/lib/dapp-bin contract.sol

  • Other Docker commands not necessary

    docker network ps
    docker network rm pyethfinality_default
    

    • References:
      • https://docs.docker.com/compose/reference/up/
      • https://docs.docker.com/engine/reference/commandline/network_prune/#related-commands

  • IGNORE - Unsuccessful attempt to install Solc using Snap on Ubuntu 16.04 (or Snapd on Ubuntu 14.04).

    • Snapcraft - https://docs.snapcraft.io/core/install-ubuntu
    • Snapd - https://github.com/snapcore/snapd
    • Snapd - https://wiki.archlinux.org/index.php/Snapd
    • Systemd - https://wiki.ubuntu.com/systemd
    • Forum - https://forum.snapcraft.io/c/snapd

    apt-get install snapd
    snap --help
    ls /lib/systemd/system/snap*; echo; systemctl list-unit-files | grep snap; echo; dpkg -L snapd | grep systemd; echo; pgrep -a snap
    

    • Note: The following approach to try and install Solc did not work as I encountered the error Failed to get D-Bus connection: No connection to service manager. and could not overcome it.

      snap download solc
      systemctl status snapd.service
      systemctl enable snapd
      service snap start
      /usr/lib/snapd/snapd
      

    • Check if booting with Systemd

      ps -p 1 -o comm=
      

    • Attempt to boot with Systemd by changing the docker-compose.yml file with the following instead of command: tail -f /dev/null did not work either

      command: /lib/systemd/systemd
      

    • After trying all the steps in this section without success, I asked a question on ethereum/solidity channel of Gitter. I received a response from @MicahZoltu, who suggested adding the following to my Dockerfile, which worked. I was then able to run solc --help in the Docker VM.

      ARG SOLC_VERSION=v0.4.18
      RUN wget --quiet --output-document /usr/local/bin/solc https://github.com/ethereum/solidity/releases/download/${SOLC_VERSION}/solc-static-linux \
          && chmod a+x /usr/local/bin/solc
      

Chapter 2 - Tests

• Run PyTest in Docker shell

  python3 -m pytest .
  
  python3 -m pytest tests/
  populus deploy --chain tester --no-wait-for-sync
  

Chapter 3 - Editor / IDE

• Editor

  • Visual Studio (VS) Code

  • Setup - View Extensions - Press SHIFT+CMD+X

    • solidity
    • Solidity Extended
    • solidity-solhint

  • Compile Solidity Contract in VS Code - Press FN+F5

  • IntelliJ Debugging with breakpoints in Python

    • brew install python3
    • File > Project Structure > Project SDK > New > /Users/Ls/.pyenv/versions/3.6.3/bin/python3
      • Note: Assumes using PyEnv
    • Edit Configurations > Click (+) > Python > Select Project SDK > Ok
    • Allow use of breakpoints (add *.py file type instead of just default *.pyw)
      • Preferences (CMD+,) > Editor > File Types > Add Wildcard > *.py
    • Run > Debug 'mockchain'

Chapter 100 - Contributors, Bugs & Issues

• To report an issue, please create a Github issue with the following details:
  • Which version of Solidity you are using
  • What was the source code (if applicable)
  • Which platform are you running on
  • How to reproduce the issue
  • What was the result of the issue
  • What was the expected behaviour

Chapter 999 - Unsorted

• Setup Populus Framework - http://populus.readthedocs.io/en/latest/quickstart.html#debian-ubuntu-mint

  • Install Populus

    pip3 install populus
    export LC_ALL=C.UTF-8
    export LANG=C.UTF-8
    

  • Initialise Populus

    populus init
    

  • Compile contract into ./build directory

    populus compile
    

• TODO - Testing with PyTest

• TODO - Front-End with Truffle

• Reference

  • Finality - https://blog.ethereum.org/2016/05/09/on-settlement-finality/

• Other References

  • Cloud9 / Linux PyEthereum - https://forum.ethereum.org/discussion/comment/16737#Comment_16737
  • Cloud9 / Linux PyEthereum - http://joeysprogrammingblog.com/tag/pyethereum/

Chapter 1000 - ERRORS - macOS Unix Bash Setup Attempt

Initially the approach was to try to create a Python application on macOS in Unix Bash without a virtual machine. I used the quick start code provided at https://github.com/pipermerriam/web3.py / https://pypi.python.org/pypi/web3/ to write main.py, and setup TestRPC code in a separate terminal window using a code snippet that I had previously prepared for another project (https://github.com/ltfschoen/solidity_test). The intention was to use Py-Solc as a wrapper for Solc.js in order to be able to use the Solidity compiler to compile the sample smart contract that was written in Solidity, but this did not work and I took drastic steps to try and make it work. Web3.py would then deploy it to the TestRPC blockchain. Unfortunately I encountered errors, so I decided to try to use a virtual machine with Docker instead. If you happen to know how to overcome the error I would be grateful if you would please create an Issue and explain it.

• Setup PyEnv - https://github.com/pyenv/pyenv#homebrew-on-mac-os-x

  • Add the following to ~/.bash_profile and then run source ~/.bash_profile.

    export PATH="/Users/username/.pyenv:$PATH"
    eval "$(pyenv init -)"
    

  • Install PyEnv

    brew update
    brew install pyenv
    pyenv --help
    pyenv install 3.6.3
    pyenv local 3.6.3
    

• Setup PyEthereum

  • About

    • Pyethereum is the Python core library for Ethereum
    • Note: This is required for Python TestRPC eth-testrpc to work.

  • Install dependencies - https://github.com/ethereum/pyethereum/wiki/Developer-Notes

    brew install pkg-config libffi autoconf automake libtool openssl
    

  • Clone repo

    git clone https://github.com/ethereum/pyethereum/
    cd pyethereum
    

  • Avoid error ethereum/pyethapp#209 scrypt-1.2.0/libcperciva/crypto/crypto_aes.c:6:10: fatal error: 'openssl/aes.h' file not found

    brew update
    brew upgrade openssl
    export LDFLAGS="-L/usr/local/opt/openssl/lib"
    export CPPFLAGS="-I/usr/local/opt/openssl/include"
    env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install cryptography
    

  • Install PyEthereum dependencies

    python setup.py install
    

• Setup Test Network for blockchain in new Unix Terminal Tab

  pip install eth-testrpc;  
  rm -rf ./db;
  mkdir -p db/chaindb;
  testrpc --db './db/chaindb'
  

• Back-end

  • Web3.py - http://web3py.readthedocs.io/en/latest/quickstart.html

    • Install Web3.py

      pip install web3
      

  • Solidity Compiler

    • Install Solc

      • Install Solc.js - https://github.com/ethereum/solc-js#usage-on-the-command-line

        npm install -g solc
        solcjs --version
        

        • Troubleshooting

          • Error 1.01 - Fix error when run main.py due to this line compile_source(contract_source_code) that results in error: FileNotFoundError: [Errno 2] No such file or directory: 'solc': 'solc':

            • Fix was to rename solcjs to solc:

              cp /Users/Ls/.nvm/versions/node/v8.7.0/bin/solcjs /Users/Ls/.nvm/versions/node/v8.7.0/bin/solc
              
              solc --version
              

          • Error 1.02 - Fix error when run main.py (after fixing Error 1.01) due to this line compile_source(contract_source_code) that results in error: Error: Cannot find module 'fs-extra' ... /Users/Ls/.nvm/versions/node/v8.7.0/bin/solc:6:10

            • Fix was to create a symlink from the solc executable to the solcjs that is installed in the NPM dependencies:

              ln -sF /Users/Ls/.nvm/versions/node/v8.7.0/lib/node_modules/solc/solcjs /Users/Ls/.nvm/versions/node/v8.7.0/bin/solc
              

      • Install Py-Solc (to be used by Python as a Wrapper for Solc.js)

        • https://pypi.python.org/pypi/py-solc
        • https://github.com/ethereum/py-solc

          pip install py-solc
          

  • Create Python program in main.py

  • Run Python CLI to experiment. Use the Simple Example in the Web3.py documentation

    • Run Python program

      python ./scripts/src/main.py
      

    • Troubleshooting

      • Error encountered is:

        Traceback (most recent call last):
          File "main.py", line 28, in <module>
            compiled_sol = compile_source(contract_source_code)
          File "/Users/Ls/.pyenv/versions/3.6.3/lib/python3.6/site-packages/solc/main.py", line 106, in compile_source
            stdoutdata, stderrdata, command, proc = solc_wrapper(**compiler_kwargs)
          File "/Users/Ls/.pyenv/versions/3.6.3/lib/python3.6/site-packages/solc/utils/string.py", line 85, in inner
            return force_obj_to_text(fn(*args, **kwargs))
          File "/Users/Ls/.pyenv/versions/3.6.3/lib/python3.6/site-packages/solc/wrapper.py", line 165, in solc_wrapper
            stderr_data=stderrdata,
        solc.exceptions.SolcError: An error occurred during execution
        > command: `solc --combined-json abi,asm,ast,bin,bin-runtime,clone-bin,devdoc,interface,opcodes,userdoc`
        > return code: `1`
        > stderr:
        
        > stdout:
        Must provide a file
        

      • Note: Pyethereum and Pyethapp travis.yml files both use APT for solc.