wip, added averaging perl script

holochain · Jan 23, 2018 · 9a0eebb · 9a0eebb
1 parent 6515deb
commit 9a0eebb
Show file tree

Hide file tree

Showing 9 changed files with 152 additions and 58 deletions.
diff --git a/README.md b/README.md
@@ -41,20 +41,23 @@ Because the architectures of Blockchain/Ethereum are so different we acknowledge
 
 ## Issues that make comparison tricky
 
-### On-chain storage
-- very expensive in ETH costs, leads to non-decentralized solutions (hashes and off chain storage)
+### Gas price fluctuation
+Over the time in which we built out these benchmarks we have seen standard [gas prices](https://ethgasstation.info/index.php) fluctuate between 4-90 Gwei.  For the purposes of choosing our standard compute unit, we averaged the gas prices for the all the days of Jan 2018 before the launch of our ICO.
 
-### Read
-- free for Ethereum (no gas cost)
-- not free in Holo
+### Read vs. Write
+For both Ethereum and Holo, there is difference in real world costs for read vs. write.  In the case of  Ethereum all of the cost paid by participants has been shifted to write, i.e. to changing state, and all read transactions are free.  For Holo both read & write transactions will carry a cost, though read transactions will be significantly cheaper because they don't trigger as much activity on the network.
 
 ### Cost differences between CPU/Storage/Bandwith/Memory
+
 - location dependent
 - application dependent
 - hardware dependent
 
 ### Resilience Factor
+Different applications will set different resilience factors, and thereby consume substantially more or less computing resources.
+
 ### Gossip
+Our gossip algorithms are incomplete.
 
 ## Methodology
 
@@ -108,13 +111,13 @@ Here's the output of running the test:
 ``` shell
 
 $ cd approvedList
-$ hcdev -mdns=true -no-nat-upnp scenario -benchmarks benchmark | perl ../bench.pl
+$ perl ../benchmark.pl
 ...
 Total chain: 556.23K
-Total DHT: 6342.62K
-Total Bytes Sent: 6208.01K
-Total Gossip Sent: 125.53K
-Total CPU: 6620ms
+Total DHT: 6342.196K
+Total Bytes Sent: 6214.686K
+Total Gossip Sent: 122.416K
+Total CPU: 6082ms
 
 ```
 
@@ -129,52 +132,55 @@ In this scenario for both the Ethereum and Holochain contexts we assume the same
 
 #### Results
 
-- _Ethereum:_
-For prosposals with 1k in the proposal text:
+For proposals with 0 bytes in the proposal text:
 ``` shell
 $ truffle test
 ...
 
-Total Gas Used:9109388
+Total Gas Used:6157696
       ✓ should run the benchmarks (6568ms)
 ```
 
-For proposals with 0 bytes in the proposal text:
+- _Ethereum:_
+For prosposals with 1k in the proposal text:
 ``` shell
 $ truffle test
 ...
 
-Total Gas Used:6157696
+Total Gas Used:9109388
       ✓ should run the benchmarks (6568ms)
 ```
 
 - _Holochain:_  In this scenario the test runs for a total of 20 seconds real time.
 
-For prosposals with 1k in the proposal text:
+For prosposals with 10bytes in the proposal text:
 ``` shell
 $ cd dao
-$ hcdev -mdns=true -no-nat-upnp scenario -benchmarks benchmark | perl ../bench.pl
+$ perl ../benchmark.pl
+
 ...
-Total chain: 49.14K
-Total DHT: 1322.79K
-Total Bytes Sent: 18886.79K
-Total Gossip Sent: 11820.55K
-Total CPU: 52180ms
+Total chain: 45.22K
+Total DHT: 1316.313K
+Total Bytes Sent: 15843.621K
+Total Gossip Sent: 9274.803K
+Total CPU: 48721ms
 ```
 
-For prosposals with 10bytes in the proposal text:
+For prosposals with 1k in the proposal text:
 ``` shell
 $ cd dao
-$ hcdev -mdns=true -no-nat-upnp scenario -benchmarks benchmark | perl ../bench.pl
-...
-Total chain: 45.22K
-Total DHT: 1293.75K
-Total Bytes Sent: 5740.94K
-Total Gossip Sent: 646.28K
-Total CPU: 24490ms
+$ perl ../benchmark.pl
 
+...
+Total chain: 49.54K
+Total DHT: 1363.782K
+Total Bytes Sent: 16685.8K
+Total Gossip Sent: 9928.417K
+Total CPU: 50874ms
 ```
 
+As you see, for Holo although there is an increase in cost, as the data size increase, it isn't nearly as significant as it is for Ethereum.
+
 ### Social Media Twitter Clone
 
 In this scenario we examine the actual costs in gas incurred by users of the Ethereum social media Twitter clone [leeroy](https://leeroy.io), and we measure the compute resources used by our own Twitter clone [Clutter](https://github.com/Holochain/clutter).
@@ -186,15 +192,14 @@ In this scenario we examine the actual costs in gas incurred by users of the Eth
 - _Holochain:_ In this scenario one user, Jane, joins the network and makes a post.  Ten other users join the network, follow Jane, and retrieve her post:
 
 ``` shell
-$ hcdev -mdns=true -no-nat-upnp scenario -benchmarks followAndShare | perl ../bench.pl
+$ cd clutter
+$ perl ../benchmark.pl
 ...
 Total chain: 17.32K
-Total DHT: 672.77K
-Total Bytes Sent: 2673.31K
-Total Gossip Sent: 325.37K
-Total CPU: 11810ms
-
-
+Total DHT: 701.389K
+Total Bytes Sent: 6349.02K
+Total Gossip Sent: 3379.842K
+Total CPU: 19510ms
 ```
 
 ### Sorting
@@ -239,13 +244,27 @@ Gas [200 elements]: 1788194
 
 - _Holochain:_ For this scenario we create a holochain app which provides a function that takes a list and then creates an entry of that list sorted, which gets puts to the DHT and then read back by 10 other nodes.   This is not quite the same scenario as above because it includes reading, but that's reasonable because reading, though less expensive in the Holocain world than write, doesn't come without some cost.
 
+Here are the results from a run with the list size to sort set to 200 to match the Ethereum scenarios above.
+
 ``` shell
-cd sortArray
-hcdev -mdns=true -no-nat-upnp scenario benchmark -benchmarks | perl ../bench.pl
+$ cd sorting
+$ ../perl benchmark.pl
 ...
-Total chain: 1.83K
-Total DHT: 244.02K
-Total Bytes Sent: 813.75K
-Total CPU: 4920ms
+Total chain: 1.6K
+Total DHT: 241.42K
+Total Bytes Sent: 655.94K
+Total Gossip Sent: 121.89K
+Total CPU: 4840ms
+
+```
+
+Here are the results from a run with the list size to sort set to 20000, which simply cannot be done in the case of Ethereum because it costs more gas than the max block gas limit.
+
+``` shell
+Total chain: 96.145K
+Total DHT: 1777.201K
+Total Bytes Sent: 2182.001K
+Total Gossip Sent: 132.369K
+Total CPU: 9679ms
 
 ```
diff --git a/bench.pl b/bench.pl
@@ -19,7 +19,7 @@
 my $gossip = 0.0;
 my $cpu = 0.0;
 foreach $line ( <STDIN> ) {
-    print $line;
+#    print $line;
     chomp( $line );
 
     $chain += $1 if $line =~ /Chain growth: ([0-9.]+)K/;

diff --git a/benchmark.pl b/benchmark.pl
@@ -0,0 +1,75 @@
+=begin comment
+This is a script to run the benchmark a number of times and average the results
+example usage:
+$ cd approvedList
+$ perl ../bench.pl
+Total chain: 556.23
+    Total DHT: 6340.86
+    Total Bytes Sent: 6306.16
+    Total CPU: 3700
+
+=end comment
+=cut
+
+
+sub run {
+
+    my $x =  `hcdev -mdns=true -no-nat-upnp scenario -benchmarks benchmark`;
+
+    my @lines = split /\n/,$x;
+    my $chain = 0.0;
+    my $dht = 0.0;
+    my $net = 0.0;
+    my $gossip = 0.0;
+    my $cpu = 0.0;
+    foreach $line (@lines) {
+        #    print $line;
+        #    chomp( $line );
+
+        $chain += $1 if $line =~ /Chain growth: ([0-9.]+)K/;
+        $dht += $1 if $line =~ /DHT growth: ([0-9.]+)K/;
+        $net += $1 if $line =~ /BytesSent: ([0-9.]+)K/;
+        $gossip += $1 if $line =~ /GossipSent: ([0-9.]+)K/;
+        $cpu += $1 if $line =~ /CPU: ([0-9.]+)ms/;
+    }
+    my %values;
+    $values{'chain'} = $chain;
+    $values{'dht'} = $dht;
+    $values{'net'} = $net;
+    $values{'gossip'} = $gossip;
+    $values{'cpu'} = $cpu;
+    return \%values;
+}
+
+my $count = 10;
+my @runs;
+foreach my $i (0..$count-1) {
+    $runs[$i] = &run();
+}
+
+my $chain = 0.0;
+my $dht = 0.0;
+my $net = 0.0;
+my $gossip = 0.0;
+my $cpu = 0.0;
+
+foreach my $i (0..$count-1) {
+    my $values = $runs[$i];
+    $chain += $values->{'chain'};
+    $dht += $values->{'dht'};
+    $net += $values->{'net'};
+    $gossip += $values->{'gossip'};
+    $cpu += $values->{'cpu'};
+}
+
+$chain /= $count;
+$dht /= $count;
+$net /= $count;
+$gossip /= $count;
+$cpu /= $count;
+
+print "Total chain: ${chain}K\n";
+print "Total DHT: ${dht}K\n";
+print "Total Bytes Sent: ${net}K\n";
+print "Total Gossip Sent: ${gossip}K\n";
+print "Total CPU: ${cpu}ms\n";
diff --git a/clutter/test/followAndShare/_config.json → clutter/test/benchmark/_config.json b/clutter/test/followAndShare/_config.json → clutter/test/benchmark/_config.json
diff --git a/clutter/test/followAndShare/jane.json → clutter/test/benchmark/jane.json b/clutter/test/followAndShare/jane.json → clutter/test/benchmark/jane.json
diff --git a/clutter/test/followAndShare/joe.json → clutter/test/benchmark/joe.json b/clutter/test/followAndShare/joe.json → clutter/test/benchmark/joe.json
diff --git a/dao/test/benchmark/partya.json b/dao/test/benchmark/partya.json
@@ -14,7 +14,7 @@
             "Convey": "All partya members add a proposal",
             "Zome": "congress",
             "FnName": "newProposal",
-            "Input": {"description":"save the %partya.%clone%%","recipient":"%implementer_key%","amount":100,"votingDeadline":"Sat Dec 16 2018 08:29:35 GMT-0500 (EST)"},
+            "Input": {"description":"save the %partya.%clone%%01234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890012345678900123456789001234567890","recipient":"%implementer_key%","amount":100,"votingDeadline":"Sat Dec 16 2018 08:29:35 GMT-0500 (EST)"},
             "Output": "%h1%"
         },
         {

diff --git a/sorting/ethsort/test/sorter.js b/sorting/ethsort/test/sorter.js
@@ -37,7 +37,7 @@ contract('Soriting algorithms', function() {
         let deployedSorter = await sorter.deployed();
         Client.link("Sorter", deployedSorter.address);
         let client = await Client.new();
-        console.log(data)
+       // console.log(data)
         await sortData(client, data);
         await checkSort(client, data);
     };
@@ -88,18 +88,18 @@ contract('Soriting algorithms', function() {
         });
 
     });
+/*
 
-    /*
-     describe('Quick Sort algorithm2:', async function () {
-
-     it("should sort 100 elements with Quick Sort", async function () {
-     await testScenario(QuickSorter, genArray(331));
-     });
-     contractShouldThrow("on 332 items", async () => {
-     await testScenario(QuickSorter, genArray(332));
-     });
-     });
-     */
+    describe('Quick Sort algorithm2:', async function () {
+
+        it("should sort 331 elements with Quick Sort", async function () {
+            await testScenario(QuickSorter, genArray(331));
+        });
+        contractShouldThrow("on 332 items", async () => {
+            await testScenario(QuickSorter, genArray(332));
+        });
+    });
+*/
 
 });
 

diff --git a/sorting/test/benchmark/sorter.json b/sorting/test/benchmark/sorter.json
@@ -4,7 +4,7 @@
             "Convey": "create a sorted list",
             "Zome": "sortArray",
             "Raw": true,
-            "Input": "var i=0;var a=[];for(i=0;i<250;i++) {a.push( Math.floor(Math.random()*10000))};sortArrayCreate(a)",
+            "Input": "var i=0;var a=[];for(i=0;i<20000;i++) {a.push( Math.floor(Math.random()*10000))};sortArrayCreate(a)",
             "Output": "%h1%"
         }
     ]