Update to support DSS 0.2.10 and manage inventory in Vat (#16)

* Skipped all the tests which require interacting with contracts.  Refactored a bit of code in main.py.  Updated test.sh to spin up pymaker's MCD testchain.

* Updated main.check_cdps with pymaker API updates, got test_bite working

* Refactored test fixtures to allow use of multiple (eth) collateral types.  Got a handful of flip tests working.

* Implemented EVM sleep when connected to Parity.  Making progress on flip tests.

* Resolved issue bidding on dent phase, made progress on flip unit tests

* Got all the flip tests working in isolation.  Work remains to allow them to be tested together.

* Integrated half the flip tests using multiple auctions

* Got tests working which use uniform auction parameters.

* All flip tests now working sequentially

* Started putting flap tests together

* Got half the flap tests working

* Got all but one of the flap tests working; last one is being a real kicker (pun intended)

* Got that last flap test working, tidied some code

* Test setup for flop auctions

* Fixed issue in reserve_dai, added parameter allowing flip tests to request extra collateral.

* Got half of the flop tests working.  Resolved issue canceling out surplus and debt.

* All flop tests are now functioning properly.

* Updated documentation

* Added test cleanups with flog/kiss/heal, resolved issue flogging multiple bites.

* Implemented vow tests, and then moved them into test_flap and test_flop.  Improved test cleanup.

* Added a few more kicks needed after tightening up ttls in the dss deployment.  Updated docs after testing against a pymaker which supports dss 0.2.10.

* Implemented facilities to manage amount of Dai joined to the Vat

* Missing file from previous commit

* Added facility to check amount of Dai in vat before bidding on flip and flop auctions.  Added scripts for manual testing.

* Resolved threading issue with auctions collection, added scripts to test flop auctions.

* Now checking MKR balance before bidding on flaps.  Added manual testing scripts.

* Removed flog-and-heal calls from the test scripts; keeper handles this implicitly.  Document integration testing artifacts.

* Resolved issue where flop wouldn't happen because flog wasn't being called

* Add support for flop delay (wait).  Improve integration test scripts.

* Updated integration tests to use testchain hosted on docker hub

* Updated docs to explain significance of the sleep in static price models

* Added facility to exit collateral on shutdown.  Reworked commandline parameters to use addresses.json to simplify configuration and avoid config errors.  Improved unit testing around these changes.

* Restored ability to configure keeper such that it does not bite/flap/flop when new opportunities arise

* Improved scripts which create CDPs to draw a random yet appropriate amount of Dai.  Updated status printer to show total debt instead of lot sizes.

* Removed pointless balance checks.  Reverted back to Martin's behavior of only letting the flogger heal and flop.

* Accommodate renaming of DssDeployment.flap and .flop, resolve unit test issues.

* Updated pymaker to latest commit

* Asynchronously bite and deal

* Added facility to print flop auctions which were not bid upon

* Cleaned up some documentation

* Updated documentation and reference to addresses used for unit testing

* Accommodate pymaker changes to hope_directly(); remove debugging messages

* Added table enumerating several keepers and showing the relevant pricing pair, per code review comment

* Updated submodule to point to appropriate pymaker

README.md

@@ -3,33 +3,42 @@
 [![Build Status](https://travis-ci.org/makerdao/auction-keeper.svg?branch=master)](https://travis-ci.org/makerdao/auction-keeper)
 [![codecov](https://codecov.io/gh/makerdao/auction-keeper/branch/master/graph/badge.svg)](https://codecov.io/gh/makerdao/auction-keeper)
 
-The _DAI Stablecoin System_ incentivizes external agents, called _keepers_,
-to automate certain operations around the Ethereum blockchain.
+The _DAI Stablecoin System_ incentivizes external agents, called _keepers_, to automate certain operations around the 
+Ethereum blockchain.  The purpose of `auction-keeper` is to:
+ * Seek out opportunities and start new auctions
+ * Detect auctions started by other participants
+ * Bid on auctions by converting token prices into bids
+
+Check out the <a href="https://youtu.be/wevzK3ADEjo?t=733">July 23rd, 2019 community meeting</a> 
+for some more information about MCD auctions and the purpose of this component.
 
 `auction-keeper` can participate in `flip` (collateral sale), `flap` (MKR buy-and-burn)
 and `flop` (MKR minting) auctions. Its unique feature is the ability to plug in external
 _bidding models_, which tell the keeper when and how high to bid. This keeper can be safely
-left running in background. The moment it notices a new auction it will spawn a new instance
+left running in background. The moment it notices or starts a new auction it will spawn a new instance
 of a _bidding model_ for it and then act according to its instructions. _Bidding models_ will
 be automatically terminated by the keeper the moment the auction expires.  The keeper also
 automatically `deal`s expired auctions if it's us who won them.
 
-Bear in mind that this keeper is still **early work in progress**. Many of the things described
-here may still change.
+This keeper is intended to be a reference implementation.  It may be used as-is, or pieces borrowed to 
+develop your own auction trading bot.  Status as of 2019.08.23:
+ * Supports DSS 0.2.10
+ * Unit testing completed
+ * Testchain integration testing completed
+ * Kovan integration testing not performed
 
 <https://chat.makerdao.com/channel/keeper>
 
 
-## Overall architecture
+## Architecture
 
-`auction-keeper` is responsible for directly interacting with `Flipper`, `Flapper` and `Flopper` auction contracts
-deployed to the Ethereum blockchain. It it responsible for querying and monitoring the current auction
-state, and also for sending all Ethereum transactions. At the same time all all decision making
-is delegated to _bidding models_.
+`auction-keeper` directly interacts with `Flipper`, `Flapper` and `Flopper` auction contracts
+deployed to the Ethereum blockchain. Decisions which involve pricing are delegated to _bidding models_.
 
 _Bidding models_ are simple processes, external to the main `auction-keeper` process. As they do not have to know
 anything about blockchain and smart contracts, they can be implemented in basically any programming language.
-The only thing they need to do is to read and write JSON documents they exchange with `auction-keeper`.
+The only thing they need to do is to read and write JSON documents they exchange with `auction-keeper`. The simplest 
+example of a bidding model is a shell script which echoes a fixed price.
 
 
 ### Monitoring ongoing auctions and discovering new ones
@@ -38,16 +47,9 @@ The main task of this keeper, as already outlined above, is to constantly monito
 discover new ones, ensure that an instance of _bidding model_ is running for each auction, provide
 these instances of the current status of their auctions and bid according to decisions taken by them.
 
-The way the auction discovery and monitoring mechanism works at the moment is pretty lame. It basically
-operates as a loop which kicks in on every new block and simply enumerates all auctions from `1` to `kicks`.
-Even if the _bidding model_ decides to send a bid, it will not be processed by the keeper until the next
-iteration of that loop. We definitely plan to upgrade this mechanism with something smarter in the future,
-especially that the current approach will stop performing well the moment the number of both current
-and historical auctions will rise. The GitHub issue for it is here: <https://github.com/makerdao/auction-keeper/issues/4>.
-
-Having said that, the current lame mechanism will probably stay around for a while as we first want
-to validate the whole architecture and only start optimizing it when it becomes necessary. Ultimately,
-good responsiveness of the keeper will be essential as the auctions space will become more competitive.
+The way the auction discovery and monitoring mechanism works at the moment is simplistic for illustration purposes. 
+It basically operates as a loop which kicks in on every new block enumerating all auctions from `1` to `kicks`.
+Bidding models are checked every 2 seconds and submitted where appropriate.
 
 
 ### Starting and stopping _bidding models_
@@ -100,7 +102,7 @@ process and tries to parse them as JSON documents. Then it extracts two fields f
   the model is willing to bid.
 * `gasPrice` (optional) - gas price in Wei to use when sending the bid.
 
-Sample message send from the model to the keeper may look like:
+A sample message sent from the model to the keeper may look like:
 ```json
 {"price": "750.0", "gasPrice": 7000000000}
 ```
@@ -121,9 +123,33 @@ you can use:
 #!/usr/bin/env bash
 
 echo "{\"price\": \"750.0\"}"  # put your price here
-sleep 1000000
+sleep 60
 ```
 
+The stdout provides a price for the collateral (for `flip` auctions) or MKR (for `flap` and `flop` auctions).  The 
+sleep locks the price in place for a minute, after which the keeper will restart the price model and read a new price.  
+Consider this your price update interval.
+
+
+## Limitations
+
+* If an auction started before the keeper was started, this keeper will not participate in it until the next block 
+is mined.
+* This keeper does not explicitly handle global settlement. If global settlement occurs while a winning bid is 
+outstanding, the keeper will not request a `yank` to refund the bid.  Workaround is to call `yank` directly using 
+`seth`.
+* Some keeper functions incur gas fees regardless of whether a bid is submitted.  This includes, but is not limited to, 
+the following actions:
+  * submitting approvals
+  * adjusting the balance of surplus to debt
+  * queuing debt for auction
+  * biting a CDP or starting a flap or flop auction
+* The keeper does not check model prices until an auction exists.  As such, it will `kick`, `flap`, or `flop` in 
+response to opportunities regardless of whether or not your Dai or MKR balance is sufficient to participate.  This too 
+imposes a gas fee.
+* After procuring more Dai, the keeper must be restarted to add it to the Vat.
+
+
 
 ## Installation
 
@@ -145,7 +171,10 @@ For some known Ubuntu and macOS issues see the [pymaker](https://github.com/make
 ```
 usage: auction-keeper [-h] [--rpc-host RPC_HOST] [--rpc-port RPC_PORT]
                       [--rpc-timeout RPC_TIMEOUT] --eth-from ETH_FROM
-                      (--flipper FLIPPER | --flapper FLAPPER | --flopper FLOPPER)
+                      [--eth-key [ETH_KEY [ETH_KEY ...]]] --addresses
+                      ADDRESSES [--type {flip,flap,flop}] [--ilk ILK]
+                      [--bid-only] [--vat-dai-target VAT_DAI_TARGET]
+                      [--keep-dai-in-vat-on-exit] [--keep-gem-in-vat-on-exit]
                       --model MODEL [--debug]
 
 optional arguments:
@@ -155,13 +184,75 @@ optional arguments:
   --rpc-timeout RPC_TIMEOUT
                         JSON-RPC timeout (in seconds, default: 10)
   --eth-from ETH_FROM   Ethereum account from which to send transactions
-  --flipper FLIPPER     Ethereum address of the Flipper contract
-  --flapper FLAPPER     Ethereum address of the Flapper contract
-  --flopper FLOPPER     Ethereum address of the Flopper contract
+  --eth-key [ETH_KEY [ETH_KEY ...]]
+                        Ethereum private key(s) to use (e.g.
+                        'key_file=aaa.json,pass_file=aaa.pass')
+  --addresses ADDRESSES
+                        path to addresses.json from the MCD deployment
+  --type {flip,flap,flop}
+                        Auction type in which to participate
+  --ilk ILK             Name of the collateral type for a flip keeper (e.g.
+                        'ETH-B', 'ZRX-A')
+  --bid-only            Do not take opportunities to create new auctions
+  --vat-dai-target VAT_DAI_TARGET
+                        Amount of Dai to keep in the Vat contract (e.g. 2000)
+  --keep-dai-in-vat-on-exit
+                        Retain Dai in the Vat on exit, saving gas when
+                        restarting the keeper
+  --keep-gem-in-vat-on-exit
+                        Retain collateral in the Vat on exit
   --model MODEL         Commandline to use in order to start the bidding model
   --debug               Enable debug output
 ```
 
+To participate in all auctions, a separate keeper must be configured for `flip` of each collateral type, as well as 
+one for `flap` and another for `flop`.  Collateral types (`ilk`s) combine the name of the token and a letter 
+corresponding to a set of risk parameters.  For example, `ETH-A` and `ETH-B` are two different collateral types for the 
+same underlying token (WETH).  
+
+![example list of keepers](README-keeper-config-example.png)
+
+Please note **collateral types in the table above are provided for illustrative purposes, and should not be interpreted 
+as an endorsement of which collaterals should be deployed to mainnet**, which will be determined by an appropriate 
+governance process.  A complete list of `ilk`s for a deployment may be gleaned from the `addresses.json`.
+
+
+### Accounting
+
+Auction contracts exclusively interact with Dai (for all auctions) and collateral (for `flip` auctions) in the `Vat`. 
+More explicitly:
+ * Dai used to bid on auctions is withdrawn from the `Vat`.
+ * Collateral and surplus Dai won at auction is placed in the `Vat`.
+
+By default, all Dai and collateral in your `eth-from` account is `exit`ed from the Vat and added to your token balance 
+when the keeper is shut down.  This feature may be disabled using the `keep-dai-in-vat-on-exit` and 
+`keep-gem-in-vat-on-exit` switches respectively.  Using an `eth-from` account with an open CDP is discouraged, 
+as debt will hinder the auction contracts' ability to access your Dai, and `auction-keeper`'s ability to `exit` Dai 
+from the `Vat`.
+
+When running multiple keepers using the same account, the balance of Dai in 
+the `Vat` will be shared across keepers.  If using the feature, set `--vat-dai-target` to the same value on each 
+keeper, and sufficiently high to cover total desired exposure.
+
+MKR used to bid on `flap` auctions is directly withdrawn from your token balance.  MKR won at `flop` auctions is 
+directly deposited to your token balance.
+
+
+## Testing
+
+This project uses [pytest](https://docs.pytest.org/en/latest/) for unit testing.  Testing depends upon on a Dockerized 
+local testchain included in `lib\pymaker\tests\config`.
+
+In order to be able to run tests, please install development dependencies first by executing:
+```
+pip3 install -r requirements-dev.txt
+```
+
+You can then run all tests with:
+```
+./test.sh
+```
+
 
 ## License
 

auction_keeper/logic.py

@@ -62,6 +62,7 @@ def __init__(self, flipper: Optional[Address], flapper: Optional[Address], flopp
         assert isinstance(flipper, Address) or (flipper is None)
         assert isinstance(flapper, Address) or (flapper is None)
         assert isinstance(flopper, Address) or (flopper is None)
+        assert isinstance(flipper, Address) or isinstance(flapper, Address) or isinstance(flopper, Address)
         assert isinstance(model_factory, ModelFactory)
 
         self.auctions = {}