v1.1.2, "scribbled notes on red pages"

- pick up latest mysql-connector-j, fixes #369
- fix an issue where maxwell could skip ahead positions if a leader failed.
- rework buffering code to be much kinder to the GC and JVM heap in case
  of very large transactions / rows inside transactions
- kinder, gentler help text when you specify an option incorrectly

Dockerfile

@@ -1,5 +1,5 @@
 FROM java:openjdk-7-jre
-ENV MAXWELL_VERSION 1.1.1
+ENV MAXWELL_VERSION 1.1.2
 
 RUN apt-get update && apt-get -y upgrade
 

README.md

@@ -16,7 +16,7 @@ but differentiates itself with these features:
 Maxwell is intended as a source for event-based readers, eg various ETL applications, search indexing,
 stat emitters.
 
-- Download: [https://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz](https://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz)
+- Download: [https://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz](https://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz)
 - Source: [https://github.com/zendesk/maxwell](https://github.com/zendesk/maxwell)
 
 <br style="clear:both"/>

docs/docs/quickstart.md

@@ -1,13 +1,13 @@
 ### Download
 ***
-- Download binary distro: [https://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz](https://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz)
+- Download binary distro: [https://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz](https://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz)
 - Sources and bug tracking is available on github: [https://github.com/zendesk/maxwell](https://github.com/zendesk/maxwell)
 - Obligatory copy/paste to terminal:
 
 ```
-curl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz \
+curl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz \
        | tar zxvf -
-cd maxwell-1.1.1
+cd maxwell-1.1.2
 ```
 
 ### Row based replication

docs/site/index.html

@@ -177,7 +177,7 @@ <h2>Maxwell = Mysql + Kafka</h2>
 <p>Maxwell is intended as a source for event-based readers, eg various ETL applications, search indexing,
 stat emitters.</p>
 <ul>
-<li>Download: <a href="https://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz">https://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz</a></li>
+<li>Download: <a href="https://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz">https://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz</a></li>
 <li>Source: <a href="https://github.com/zendesk/maxwell">https://github.com/zendesk/maxwell</a></li>
 </ul>
 <p><br style="clear:both"/></p>
@@ -187,7 +187,7 @@ <h2>Maxwell = Mysql + Kafka</h2>
     &quot;table&quot;: &quot;maxwell&quot;,
     &quot;type&quot;: &quot;insert&quot;,
     &quot;ts&quot;: 1449786310,
-    &quot;xid&quot;: 940753,
+    &quot;xid&quot;: 940752,
     &quot;commit&quot;: true,
     &quot;data&quot;: { &quot;id&quot;:1, &quot;daemon&quot;: &quot;Stanislaw Lem&quot; }
   }

docs/site/mkdocs/search_index.json

@@ -2,17 +2,17 @@
     "docs": [
         {
             "location": "/", 
-            "text": "Maxwell = Mysql + Kafka\n\n\n\n\n\nThis is Maxwell's daemon, an application that reads MySQL binlogs and writes row updates to Kafka as JSON.\nIt's playing in the same space as \nmypipe\n and \ndatabus\n,\nbut differentiates itself with these features:\n\n\n\n\nWorks with an unpatched mysql\n\n\nParses ALTER/CREATE/DROP table statements, which allows Maxwell to always have a correct view of the mysql schema\n\n\nStores its replication position and needed data within the mysql server itself\n\n\nRequires no external dependencies (save Kafka, if used)\n\n\nEschews the complexity of Avro for plain old JSON.\n\n\nMinimal setup\n\n\n\n\nMaxwell is intended as a source for event-based readers, eg various ETL applications, search indexing,\nstat emitters.\n\n\n\n\nDownload: \nhttps://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz\n\n\nSource: \nhttps://github.com/zendesk/maxwell\n\n\n\n\n\n\n  mysql\n insert into `test`.`maxwell` set id = 1, daemon = 'Stanislaw Lem';\n  maxwell: {\n    \ndatabase\n: \ntest\n,\n    \ntable\n: \nmaxwell\n,\n    \ntype\n: \ninsert\n,\n    \nts\n: 1449786310,\n    \nxid\n: 940753,\n    \ncommit\n: true,\n    \ndata\n: { \nid\n:1, \ndaemon\n: \nStanislaw Lem\n }\n  }\n\n\n\n\n  mysql\n update test.maxwell set daemon = 'firebus!  firebus!' where id = 1;\n  maxwell: {\n    \ndatabase\n: \ntest\n,\n    \ntable\n: \nmaxwell\n,\n    \ntype\n: \nupdate\n,\n    \nts\n: 1449786341,\n    \nxid\n: 940786,\n    \ncommit\n: true,\n    \ndata\n: {\nid\n:1, \ndaemon\n: \nFirebus!  Firebus!\n},\n    \nold\n:  {\ndaemon\n: \nStanislaw Lem\n}\n  }\n\n\n\n\n\n  jQuery(document).ready(function () {\n    jQuery(\"#maxwell-header\").append(\n      jQuery(\"<img alt='The Daemon, maybe' src='./img/cyberiad_1.jpg' id='maxwell-daemon-image'>\")\n    );\n    jQuery(\"pre\").addClass(\"home-code\");\n  });", 
+            "text": "Maxwell = Mysql + Kafka\n\n\n\n\n\nThis is Maxwell's daemon, an application that reads MySQL binlogs and writes row updates to Kafka as JSON.\nIt's playing in the same space as \nmypipe\n and \ndatabus\n,\nbut differentiates itself with these features:\n\n\n\n\nWorks with an unpatched mysql\n\n\nParses ALTER/CREATE/DROP table statements, which allows Maxwell to always have a correct view of the mysql schema\n\n\nStores its replication position and needed data within the mysql server itself\n\n\nRequires no external dependencies (save Kafka, if used)\n\n\nEschews the complexity of Avro for plain old JSON.\n\n\nMinimal setup\n\n\n\n\nMaxwell is intended as a source for event-based readers, eg various ETL applications, search indexing,\nstat emitters.\n\n\n\n\nDownload: \nhttps://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz\n\n\nSource: \nhttps://github.com/zendesk/maxwell\n\n\n\n\n\n\n  mysql\n insert into `test`.`maxwell` set id = 1, daemon = 'Stanislaw Lem';\n  maxwell: {\n    \ndatabase\n: \ntest\n,\n    \ntable\n: \nmaxwell\n,\n    \ntype\n: \ninsert\n,\n    \nts\n: 1449786310,\n    \nxid\n: 940752,\n    \ncommit\n: true,\n    \ndata\n: { \nid\n:1, \ndaemon\n: \nStanislaw Lem\n }\n  }\n\n\n\n\n  mysql\n update test.maxwell set daemon = 'firebus!  firebus!' where id = 1;\n  maxwell: {\n    \ndatabase\n: \ntest\n,\n    \ntable\n: \nmaxwell\n,\n    \ntype\n: \nupdate\n,\n    \nts\n: 1449786341,\n    \nxid\n: 940786,\n    \ncommit\n: true,\n    \ndata\n: {\nid\n:1, \ndaemon\n: \nFirebus!  Firebus!\n},\n    \nold\n:  {\ndaemon\n: \nStanislaw Lem\n}\n  }\n\n\n\n\n\n  jQuery(document).ready(function () {\n    jQuery(\"#maxwell-header\").append(\n      jQuery(\"<img alt='The Daemon, maybe' src='./img/cyberiad_1.jpg' id='maxwell-daemon-image'>\")\n    );\n    jQuery(\"pre\").addClass(\"home-code\");\n  });", 
             "title": "Overview"
         }, 
         {
             "location": "/quickstart/", 
-            "text": "Download\n\n\n\n\n\n\nDownload binary distro: \nhttps://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz\n\n\nSources and bug tracking is available on github: \nhttps://github.com/zendesk/maxwell\n\n\nObligatory copy/paste to terminal:\n\n\n\n\ncurl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz \\\n       | tar zxvf -\ncd maxwell-1.1.1\n\n\n\n\nRow based replication\n\n\n\n\nMaxwell can only operate if row-based replication is on.\n\n\n$ vi my.cnf\n\n[mysqld]\nserver-id=1\nlog-bin=master\nbinlog_format=row\n\n\n\n\nOr on a running server:\n\n\nmysql\n set global binlog_row_image=FULL;\n\n\n\n\nnote\n: When changing the binlog format on a running server, currently connected mysql clients will continue to replication in STATEMENT format --\nin order to change to row-based replication, you must reconnect all active clients to the server.\n\n\nMysql permissions\n\n\n\n\nMaxwell stores all the state it needs within the mysql server itself, in the database called specified by the \nschema_database\n option. By default the database is named \nmaxwell\n.\n\n\nmysql\n GRANT ALL on maxwell.* to 'maxwell'@'%' identified by 'XXXXXX';\nmysql\n GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE on *.* to 'maxwell'@'%';\n\n# or for running maxwell locally:\n\nmysql\n GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE on *.* to 'maxwell'@'localhost' identified by 'XXXXXX';\nmysql\n GRANT ALL on maxwell.* to 'maxwell'@'localhost';\n\n\n\n\n\nSTDOUT producer\n\n\n\n\nUseful for smoke-testing the thing.\n\n\nbin/maxwell --user='maxwell' --password='XXXXXX' --host='127.0.0.1' --producer=stdout\n\n\n\n\nIf all goes well you'll see maxwell replaying your inserts:\n\n\nmysql\n insert into test.maxwell set id = 5, daemon = 'firebus!  firebus!';\nQuery OK, 1 row affected (0.04 sec)\n\n(maxwell)\n{\ntable\n:\nmaxwell\n,\ntype\n:\ninsert\n,\ndata\n:{\nid\n:5,\ndaemon\n:\nfirebus!  firebus!\n},\nts\n: 123456789}\n\n\n\n\nKafka producer\n\n\n\n\nBoot kafka as described here:  \nhttp://kafka.apache.org/documentation.html#quickstart\n, then:\n\n\nbin/maxwell --user='maxwell' --password='XXXXXX' --host='127.0.0.1' \\\n   --producer=kafka --kafka.bootstrap.servers=localhost:9092\n\n\n\n\nThis will start writing to the topic \"maxwell\".", 
+            "text": "Download\n\n\n\n\n\n\nDownload binary distro: \nhttps://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz\n\n\nSources and bug tracking is available on github: \nhttps://github.com/zendesk/maxwell\n\n\nObligatory copy/paste to terminal:\n\n\n\n\ncurl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz \\\n       | tar zxvf -\ncd maxwell-1.1.2\n\n\n\n\nRow based replication\n\n\n\n\nMaxwell can only operate if row-based replication is on.\n\n\n$ vi my.cnf\n\n[mysqld]\nserver-id=1\nlog-bin=master\nbinlog_format=row\n\n\n\n\nOr on a running server:\n\n\nmysql\n set global binlog_row_image=FULL;\n\n\n\n\nnote\n: When changing the binlog format on a running server, currently connected mysql clients will continue to replication in STATEMENT format --\nin order to change to row-based replication, you must reconnect all active clients to the server.\n\n\nMysql permissions\n\n\n\n\nMaxwell stores all the state it needs within the mysql server itself, in the database called specified by the \nschema_database\n option. By default the database is named \nmaxwell\n.\n\n\nmysql\n GRANT ALL on maxwell.* to 'maxwell'@'%' identified by 'XXXXXX';\nmysql\n GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE on *.* to 'maxwell'@'%';\n\n# or for running maxwell locally:\n\nmysql\n GRANT SELECT, REPLICATION CLIENT, REPLICATION SLAVE on *.* to 'maxwell'@'localhost' identified by 'XXXXXX';\nmysql\n GRANT ALL on maxwell.* to 'maxwell'@'localhost';\n\n\n\n\n\nSTDOUT producer\n\n\n\n\nUseful for smoke-testing the thing.\n\n\nbin/maxwell --user='maxwell' --password='XXXXXX' --host='127.0.0.1' --producer=stdout\n\n\n\n\nIf all goes well you'll see maxwell replaying your inserts:\n\n\nmysql\n insert into test.maxwell set id = 5, daemon = 'firebus!  firebus!';\nQuery OK, 1 row affected (0.04 sec)\n\n(maxwell)\n{\ntable\n:\nmaxwell\n,\ntype\n:\ninsert\n,\ndata\n:{\nid\n:5,\ndaemon\n:\nfirebus!  firebus!\n},\nts\n: 123456789}\n\n\n\n\nKafka producer\n\n\n\n\nBoot kafka as described here:  \nhttp://kafka.apache.org/documentation.html#quickstart\n, then:\n\n\nbin/maxwell --user='maxwell' --password='XXXXXX' --host='127.0.0.1' \\\n   --producer=kafka --kafka.bootstrap.servers=localhost:9092\n\n\n\n\nThis will start writing to the topic \"maxwell\".", 
             "title": "Quick Start"
         }, 
         {
             "location": "/quickstart/#download", 
-            "text": "Download binary distro:  https://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz  Sources and bug tracking is available on github:  https://github.com/zendesk/maxwell  Obligatory copy/paste to terminal:   curl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz \\\n       | tar zxvf -\ncd maxwell-1.1.1", 
+            "text": "Download binary distro:  https://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz  Sources and bug tracking is available on github:  https://github.com/zendesk/maxwell  Obligatory copy/paste to terminal:   curl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz \\\n       | tar zxvf -\ncd maxwell-1.1.2", 
             "title": "Download"
         }, 
         {

docs/site/quickstart/index.html

@@ -177,13 +177,13 @@
 <h3 id="download">Download</h3>
 <hr />
 <ul>
-<li>Download binary distro: <a href="https://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz">https://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz</a></li>
+<li>Download binary distro: <a href="https://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz">https://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz</a></li>
 <li>Sources and bug tracking is available on github: <a href="https://github.com/zendesk/maxwell">https://github.com/zendesk/maxwell</a></li>
 <li>Obligatory copy/paste to terminal:</li>
 </ul>
-<pre><code>curl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.1.1/maxwell-1.1.1.tar.gz \
+<pre><code>curl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.1.2/maxwell-1.1.2.tar.gz \
        | tar zxvf -
-cd maxwell-1.1.1
+cd maxwell-1.1.2
 </code></pre>
 
 <h3 id="row-based-replication">Row based replication</h3>

docs/site/sitemap.xml

@@ -4,55 +4,55 @@
 
     <url>
      <loc>None/</loc>
-     <lastmod>2016-05-23</lastmod>
+     <lastmod>2016-06-27</lastmod>
      <changefreq>daily</changefreq>
     </url>
 
 
 
     <url>
      <loc>None/quickstart/</loc>
-     <lastmod>2016-05-23</lastmod>
+     <lastmod>2016-06-27</lastmod>
      <changefreq>daily</changefreq>
     </url>
 
 
 
     <url>
      <loc>None/config/</loc>
-     <lastmod>2016-05-23</lastmod>
+     <lastmod>2016-06-27</lastmod>
      <changefreq>daily</changefreq>
     </url>
 
 
 
     <url>
      <loc>None/kafka/</loc>
-     <lastmod>2016-05-23</lastmod>
+     <lastmod>2016-06-27</lastmod>
      <changefreq>daily</changefreq>
     </url>
 
 
 
     <url>
      <loc>None/dataformat/</loc>
-     <lastmod>2016-05-23</lastmod>
+     <lastmod>2016-06-27</lastmod>
      <changefreq>daily</changefreq>
     </url>
 
 
 
     <url>
      <loc>None/bootstrapping/</loc>
-     <lastmod>2016-05-23</lastmod>
+     <lastmod>2016-06-27</lastmod>
      <changefreq>daily</changefreq>
     </url>
 
 
 
     <url>
      <loc>None/compat/</loc>
-     <lastmod>2016-05-23</lastmod>
+     <lastmod>2016-06-27</lastmod>
      <changefreq>daily</changefreq>
     </url>
 

pom.xml

@@ -4,7 +4,7 @@
 
   <groupId>com.zendesk</groupId>
   <artifactId>maxwell</artifactId>
-  <version>1.1.1</version>
+  <version>1.1.2</version>
   <packaging>jar</packaging>
 
   <name>maxwell</name>