added base64 file transfer encoding with hex fallback #34

AndiDittrich · Apr 25, 2017 · 808b60b · 808b60b
1 parent c7342e1
commit 808b60b
Show file tree

Hide file tree

Showing 8 changed files with 377 additions and 27 deletions.
diff --git a/CHANGES.md b/CHANGES.md
@@ -1,5 +1,9 @@
+### 2.1.0 ###
+* Added: Support for native [encode](http://nodemcu.readthedocs.io/en/master/en/modules/encoder/) module to use **base64** as transfer encoding (speed up the transfer by factor 4) - suggested by [MickaelGuilloux on GitHub](https://github.com/AndiDittrich/NodeMCU-Tool/pull/34) #32 
+* Bugfix: By uploading multiple files the transfer-encoder functions where uploaded each time
+
 ### 2.0.4 ###
-* Bugfix: Download command failed because of API changes in NodeMCU Firmware **v1.5.4** - thanks to [verneroberts on GitHub](https://github.com/AndiDittrich/NodeMCU-Tool/pull/24) #42
+* Bugfix: Download command failed because of API changes in NodeMCU Firmware **v1.5.4** - thanks to [verneroberts on GitHub](https://github.com/AndiDittrich/NodeMCU-Tool/pull/24) #24
 
 ### 2.0.3 ###
 * Bugfix: Terminal in **non-TTY** mode caused fatal error on connect - thanks to [stephanMettler on GitHub](https://github.com/AndiDittrich/NodeMCU-Tool/issues/23) #23

diff --git a/README.md b/README.md
@@ -48,6 +48,7 @@ Related Documents
 * [Common Use-Cases and Examples](docs/Examples.md)
 * [Programmatic Usage](docs/ProgrammaticUsage.md)
 * [Fixing Reset-on-Connect Issue](docs/Reset_on_Connect.md)
+* [File Transfer Encoding](docs/TransferEncoding.md)
 * [Webstorm Integration](docs/WebstormIntegration.md)
 * [Contribution Guidelines](CONTRIBUTE.md)
 * [NodeMCU DEVKIT Schematics](https://github.com/nodemcu/nodemcu-devkit-v1.0/blob/master/NODEMCU_DEVKIT_V1.0.PDF)
@@ -184,6 +185,8 @@ $ nodemcu-tool mkfs --port=/dev/ttyUSB0
 
 ### 4. Upload a new File ###
 
+**Hint** include the native [encoder Module](http://nodemcu.readthedocs.io/en/master/en/modules/encoder/) into your firmware to speed-up the uploading by factor 4..10!
+
 ```shell
 $ nodemcu-tool upload --port=/dev/ttyUSB0 --optimize helloworld.lua
 [NodeMCU-Tool] Connected
@@ -324,6 +327,8 @@ By default, the serial connection uses a 9600 baud with 8N1 - this means maximal
 Due to the limitations of a line-wise file upload, these maximal transfer rate cannot be reached, because every line has to be processed by the lua interpreter and NodeMCU-Tool is waiting for it's response.
 It's recommended to use the `--optimize` flag to strip whitespaces before uploading. Additionally, newer firmware versions `1.x.x` using an auto-baudrate detection algorithm - this means you can increase the baudrate to e.g. 115200 `--baud 115200` to speed up the transfer 
 
+Additionally include the native [encoder Module](http://nodemcu.readthedocs.io/en/master/en/modules/encoder/) into your firmware to speed-up the uploading by factor 4..10!
+
 Any Questions ? Report a Bug ? Enhancements ?
 ---------------------------------------------
 Please open a new issue on [GitHub](https://github.com/AndiDittrich/NodeMCU-Tool/issues)

diff --git a/docs/TransferEncoding.md b/docs/TransferEncoding.md
@@ -0,0 +1,83 @@
+File Transfer Encoding
+==========================
+
+NodeMCU-Tool uses **hex** or **base64** encoding to transfer the files binary safe.
+
+To use the **base64** mode, you have to include the native [encode](http://nodemcu.readthedocs.io/en/master/en/modules/encoder/) module into your firmware - this will speed up the file transfer by factor 4..10!
+
+File Upload Helper Function
+-----------------------------
+
+The following lua code is used to decode the stream on the NodeMCU Module. In case the `encoder.fromBase64` function is available it will be used.
+
+**Original Version**
+
+
+```lua
+-- Base64 Module available ?
+if encoder and encoder.fromBase64 then
+    -- Use build-in BASE64 Decoder
+    _G.__nmtwrite = function(s) 
+        file.write(encoder.fromBase64(s))
+    end
+    print("b")
+else
+    -- Use classic HEX Decoder
+    _G.__nmtwrite = function(s) 
+        for c in s:gmatch('..') do 
+            file.write(string.char(tonumber(c, 16))) 
+        end 
+    end
+    print("h") 
+end
+```
+
+**Compressed Version**
+
+```lua
+if encoder and encoder.fromBase64 then _G.__nmtwrite = function(s) file.write(encoder.fromBase64(s)) 
+end print('b') else _G.__nmtwrite = function(s) for c in s:gmatch('..') 
+do file.write(string.char(tonumber(c, 16))) end end print('h') end
+```
+
+
+File Download Helper Function
+-----------------------------
+
+The following lua code is used to encode the stream on the NodeMCU Module. In case the `encoder.toBase64` function is available it will be used.
+
+**Original Version**
+
+
+```lua
+function __nmtread()
+    -- flag to indicate base64 encoding
+    local b64 = encoder and encoder.toBase64
+
+    while true do
+
+        -- Base64 Chunk length (without padding): 6bits per character => for n bytes 4*(n/3) chars required
+        c = file.read(b64 and 240 or 1)
+
+        -- No more data available
+        if c == nil then 
+            print('') 
+            break 
+        end 
+
+        -- Base64 or hex encoding ?
+        uart.write(0, b64 and encoder.toBase64(c) or string.format('%02X', string.byte(c)))
+    end
+
+    -- final delimiter
+    print('')
+end
+```
+
+**Compressed Version**
+
+```lua
+function __nmtread()local b = encoder and encoder.toBase64 while true do c = file.read(b and 240 or 1) 
+if c==nil then print('')break end uart.write(0, b and encoder.toBase64(c) 
+or string.format('%02X', string.byte(c)))end print('') end
+```
diff --git a/lib/LuaCommandBuilder.js b/lib/LuaCommandBuilder.js
@@ -36,11 +36,11 @@ var lua_commands = {
     // file close & flush
     fileCloseFlush: 'file.flush() file.close()',
 
-    // helper function to write hex encoded content to file
-    hexWriteHelper: "function __hexwrite(s) for c in s:gmatch('..') do file.write(string.char(tonumber(c, 16))) end end",
+    // helper function to write hex/base64 encoded content to file @see docs/TransferEncoding.md
+    transferWriteHelper: "if encoder and encoder.fromBase64 then _G.__nmtwrite = function(s) file.write(encoder.fromBase64(s)) end print('b') else _G.__nmtwrite = function(s) for c in s:gmatch('..') do file.write(string.char(tonumber(c, 16))) end end print('h') end",
 
-    // helper function to read file as hex and write content to uart
-    hexReadHelper: "function __hexread() while true do c = file.read(1) if c == nil then print('') break end uart.write(0, string.format('%02X', string.byte(c))) end end"
+    // helper function to read hex/base64 encoded content from file  @see docs/TransferEncoding.md
+    transferReadHelper: "function __nmtread()local b = encoder and encoder.toBase64 while true do c = file.read(b and 240 or 1) if c==nil then print('')break end uart.write(0, b and encoder.toBase64(c) or string.format('%02X', string.byte(c)))end print('') end"
 };
 
 // prepare command be escaping args

diff --git a/lib/NodeMCU-Tool.js b/lib/NodeMCU-Tool.js
@@ -160,9 +160,6 @@ Tool.upload = function(localFiles, options, onProgess){
         // display filename
         _logger.log('Uploading "' + localFile + '" >> "' + remoteFilename + '"...');
 
-        // trigger a progress update
-        onProgess(0, fileInfo.size, fileUploadIndex);
-
         // normalize the remote filename (strip relative parts)
         remoteFilename = remoteFilename.replace(/\.\.\//g, '').replace(/\.\./g, '').replace(/^\.\//, '');
 

diff --git a/lib/NodeMcuConnector.js b/lib/NodeMcuConnector.js
@@ -16,7 +16,8 @@ function NodeMcuConnector(devicename, baudrate){
     this.isConnected = false;
     this.name = devicename;
     this.baudrate = baudrate;
-    this.isHexWriteHelperUploaded = false;
+    this.isTransferWriteHelperUploaded = false;
+    this.transferEncoding = 'hex';
 
     // handle low level errors
     this.device.onError(function(err){
@@ -211,17 +212,17 @@ NodeMcuConnector.prototype.upload = function(localName, remoteName, options, com
         rawContent = _luaOptimizer.optimize(rawContent);
     }
 
-    // convert buffer to hex
-    var content = rawContent.toString('hex');
+    // wrapper to start the transfer
+    var startTransfer = function(){
+        // convert buffer to hex or base64
+        var content = rawContent.toString(this.transferEncoding);
 
-    // get absolute filesize
-    var absoluteFilesize = rawContent.length;
+        // get absolute filesize
+        var absoluteFilesize = content.length;
 
-    // split file content into chunks
-    var chunks = content.match(/.{1,232}/g);
+        // split file content into chunks
+        var chunks = content.match(/.{1,232}/g);
 
-    // wrapper to start the transfer
-    var startTransfer = function(){
         // open remote file for write
         this.device.executeCommand(_luaCommandBuilder.prepare('fileOpen', [remoteName, 'w+']), function(err, echo, response){
             // successful opened ?
@@ -241,10 +242,10 @@ NodeMcuConnector.prototype.upload = function(localName, remoteName, options, com
                     var l = chunks.shift();
 
                     // increment size counter
-                    currentUploadSize += l.length/2 ;
+                    currentUploadSize += l.length;
 
                     // write first element to file
-                    this.device.executeCommand('__hexwrite("' + l + '")', function(err, echo, response){
+                    this.device.executeCommand('__nmtwrite("' + l + '")', function(err, echo, response){
                         if (err){
                             completeCb('Cannot write chunk to remote file - ' + err, null);
                             return;
@@ -278,26 +279,38 @@ NodeMcuConnector.prototype.upload = function(localName, remoteName, options, com
 
 
     // hex write helper already uploaded within current session ?
-    if (this.isHexWriteHelperUploaded){
+    if (this.isTransferWriteHelperUploaded){
         // start transfer directly
         startTransfer();
 
     // otherwise upload helper
     }else{
         // transfer helper function to decode hex data
-        this.device.executeCommand(_luaCommandBuilder.command.hexWriteHelper, function(err, echo, response) {
+        this.device.executeCommand(_luaCommandBuilder.command.transferWriteHelper, function(err, echo, response) {
             // successful opened ?
             if (err) {
                 completeCb('Cannot transfer hex.decode helper function - ' + err);
                 return;
             }
 
+            // get transfer encoding
+            if (response == 'b'){
+                this.transferEncoding = 'base64'
+            }else if (response == 'h'){
+                this.transferEncoding = 'hex'
+            }else{
+                complteCb('Unknown transfer encoding - ' + response);
+            }
+
+            // show encoding
+            _logger.log('Transfer-Mode: ' + this.transferEncoding);
+
             // set flag
-            this.isHexWriteHelperUploaded = true;
+            this.isTransferWriteHelperUploaded = true;
 
             // start file transfer on upload complete
             startTransfer();
-        });
+        }.bind(this));
     }
 };
 
@@ -461,7 +474,7 @@ NodeMcuConnector.prototype.download = function(remoteName, cb){
     }
 
     // transfer helper function to encode hex data
-    this.device.executeCommand(_luaCommandBuilder.command.hexReadHelper, function(err, echo, response) {
+    this.device.executeCommand(_luaCommandBuilder.command.transferReadHelper, function(err, echo, response) {
         // successful opened ?
         if (err) {
             cb('Cannot transfer hex.encode helper function - ' + err);
@@ -477,14 +490,18 @@ NodeMcuConnector.prototype.download = function(remoteName, cb){
             }
 
             // write first element to file
-            this.device.executeCommand('__hexread()', function(err, echo, filecontent){
+            this.device.executeCommand('__nmtread()', function(err, echo, filecontent){
                 if (err){
                     cb('Cannot read remote file content - ' + err, null);
                     return;
                 }
 
-                // decode file content
-                var data = new Buffer(filecontent, 'hex');
+                // encoding
+                var tEncoding = filecontent.match(/^[0-9A-F]+$/gi) ? 'hex' : 'base64';
+                _logger.log('Transfer-Encoding: ' + tEncoding);
+
+                // decode file content + detect encoding
+                var data = new Buffer(filecontent, tEncoding);
 
                 // send file close command
                 this.device.executeCommand(_luaCommandBuilder.command.fileClose, function(err, echo, response){