Switch branches/tags
Nothing to show
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
..
Failed to load latest commit information.
docimages
node-red
node
sample.data
src
web
README.md
Two.RPi.Zero.fzz
adafruit.io.properties
autoparse.sh
build.gradle
casestudy.md
cc.properties
char.console.properties
demos.md
dp_2011_04_15.csv
examples.md
go
killmux
killsf
logging.properties
loraRF95.properties
mqtt.properties
mux.sh
mux.sunflower.sh
nmea.mux.2.properties
nmea.mux.3.properties
nmea.mux.4.properties
nmea.mux.5.properties
nmea.mux.6.properties
nmea.mux.7.properties
nmea.mux.8.properties
nmea.mux.9.properties
nmea.mux.gps.log.properties
nmea.mux.gps.log.small.properties
nmea.mux.in.out.properties
nmea.mux.lsm303.properties
nmea.mux.properties
nmea.mux.replay.log.properties
nmea.mux.rpi.2.demo.properties
nmea.mux.rpi.demo.properties
nmea.mux.substitute.demo.properties
nmea.mux.sun.flower.02.properties
nmea.mux.sun.flower.03.properties
nmea.mux.sun.flower.demo.properties
nmea.mux.sun.flower.properties
nmea.mux.weather.station.properties
nmea.mux.weather.station.tcp.properties
package.json
rc.local
rmi.client.sh
ssd1306.fzz
ssd1306.properties
sunflower.properties
sunflower.rest.properties
test.aio
tomux
tosf
weather.station.properties
wsnmea.js
wsnmea.parser.js
zero-deviation.csv

README.md

NMEA Multiplexer

NMEA channels management, in and out.


NMEA (National Marine Electronics Association) is one of the oldest IT standards, defining how sensor data should be conveyed.


Any input (File, Serial, TCP, UDP, WebSocket, Sensors, Computations, ...), any output (File, Serial, TCP, UDP, WebSockets...), and a REST API on top of that.

Designed to run on very small boards, like a Raspberry PI Zero, and with possibly no Internet access.

Warning: It is not designed to scale as an enterprise server!

The operations on the Serial port require libRxTx. This is included in the gradle dependencies. To be able to use it outside gradle, run (on Linux/Debian/Raspberry PI):

 sudo apt-get install librxtx-java

See how this is used and referred to in mux.sh.


For the impatient: go here to get started now. The others will keep reading.


Includes

  • NMEA Strings Parser
  • NMEA Strings generator
  • Serial port Reader / Writer
  • TCP Reader / Writer
  • UDP Reader / Writer
  • WebSocket client (read/write)

Also includes

  • A tiny (but amazing) HTTP REST server, for admin purpose.

Javadoc

Javadoc can be generated by using

 ../gradlew javadoc

The generated files will be in build/docs/javadoc. Open the index.html in your browser.

Some definitions

At the center of the picture, there is the Multiplexer (aka MUX).

The MUX reads from channels and writes to forwarders.

A channel is an NMEA data provider, a forwarder is an NMEA data producer, it can be used as input from another Multiplexer.

Just to insist: A channel is an NMEA data provider. This means that it produces (pure) NMEA Sentences!! This is pure NMEA. Nothing fancy like a JSON Object or whatever object representing the same data in another format. NMEA is already a well known and well defined format.

In addition, we can have sensors. A sensor is reading data from a transducer, and produces NMEA sentences read by the Mux. In other words, a sensor is talking to the Multiplexer, and can be seen as a channel.

Also, a computer is using NMEA data collected by the Multiplexer to produce other NMEA data that will be broadcasted by the forwarders. For example, True Wind computed with Apparent Wind data and the GPS data.

Note: to compute the required data, we have a cache, where the data required by the computers are pushed. This cache is initialized before starting the computers, with parameters contained in the properties file used at startup.

This cache is necessary to perform damping and smoothing operations - among others.

Finally, we have tranformers, that transform NMEA data into another (proprietary) format, and then behave like a a regular forwarder to provide them to whoever is interested. A transformer is also a forwarder. See below examples of transformers.

Examples

Channels:

  • Serial reads NMEA data from a Serial Port
  • TCP reads NMEA data from a TCP server
  • WebSocket reads NMEA data from a WebSocket server (this is a WebSocket client)
  • File reads NMEA data from a log file
  • etc

Forwarders:

  • Serial writes NMEA data to a Serial Port (not implemented yet)
  • TCP writes NMEA data to a TCP port
  • WebSocket writes NMEA data to a WebSocket server (this is also a WebSocket client)
  • File write NMEA data to a log file
  • etc

Sensors:

  • BME280 reads raw data from a BME280 sensor (Temperature, Pressure, Humidity), and produces XDR and MDA NMEA Sentences.
  • BMP180 reads raw data from a BMP180 sensor (Temperature, Pressure), and produces XDR and MDA NMEA Sentences.
  • HTU21D-F reads raw data from a HTU21D-F sensor (Temperature, Humidity), and produces XDR NMEA Sentences.
  • LSM303 reads raw data from a LSM303 sensor (3D magnetometer and accelerometer), and produces XDR NMEA Sentences for pitch and roll.
  • ... and the list is not closed. I2C Wiring

Computers:

  • True Wind computer (produces MDA, MWD data)
  • Current computer (produces VDR data)

Important: Computers require the cache to be initialized. This is set in the properties file:

 init.cache=true

Transformers:

  • GPSD data (See this)
  • Custom data
A word about the Current Computer

A basic approach to compute the current would be to do it by instant triangulation, figuring the Course Made Good (CMG) and comparing it with the GPS Data (Course and Speed Over Ground). A better approach turned out to compute the current over a given period of time. For example, you can perform this calculation by comparing the position you should be at with the CMG only (i.e. as if there was no current) and the one given by the GPS, over periods like 30 seconds, 1 minute, 10 minutesFSupporte, etc, using a smoothing of the Boat Speed (BSP) and the CMG. The cache is designed to manage several such computations in parallel, they are discriminated by the length of their time-buffer (30 seconds, 5 minutes, etc). The accuracy of such a computations is much higher than the instant triangulation. See this article for details.

Overview

Overall Overview

Note: There is no Transformer on the picture above

Note

There is an rmi forwarder. This is a work in progress, but it works. It is feeding an RMI server that can then be accessed by an RMI client. See an example of such a client in samples.rmi.client.SampleRMIClient.

To see it at work (aka Get Started)

See the class nmea.mux.GenericNMEAMultiplexer, it uses the file nmea.mux.properties to define what to read, and what to re-broacdast it to. See it to understand its content (should be clear enough).

Those settings can be modified once the mux is started, throught the REST API.

To compile and build:

 $> ../gradlew --daemon shadowJar

To run it, modify mux.sh to fit your environment, and run

 $> ./mux.sh
Filtering

The Channels - aka Consumers - support sentence filtering. Filtering can be applied on Device ID, and on Sentence ID. Filters can be positive or negative.

Positive filters are linked with and or, Negative ones with an and.

A (sentence) filter like "HDM", "GLL", "~RMC", "~XDR" would mean

(HDM or GLL) and (not RMC and not XDR)

It is the user's responsibility not to have contradiction in the filters, like [ "GLL", "~GLL" ], no verification is done in this area.

Note: This is just providing the possibility to negate an expression. Convenient, but limited. The best would probably be to use regular expressions (RegExp). Big drawback though: for the majority of the users, the RegExp syntax could be complex, too complex, or even scary...

About transformers

There is an example of a transformer in WebSocketProcessor.java. As you would see, it is just implementing the Forwarder interface, and this is where it fits in the picture above. A Transformer is just reworking the data before forwarding them as a regular forwarder would.

The example in WebSocketProcessor.java is transforming the NMEA Data in the format expected by a Pebble (this is a smart watch) application. See it here. Data are expected as a json object, over WebSocket. The expected data look like:

{
    "gpstime": 1290377346000,
    "gpsdatetime": 1290377346000,
    "wp": "RANGI   ",
    "d2wp": 561.6,
    "b2wp": 230,
    "xte": 3.0,
    "lat": -9.1102,
    "lng": -140.21108333333333,
    "gpssolardate": 1290343695340,
    "log": 3013.0,
    "daylog": 12.4,
    "cog": 218,
    "sog": 7.2,
    "awa": -121,
    "aws": 17.8,
    "dbt": 1.7000000476837158,
    "hdg": 220,
    "bsp": 6.6,
    "wtemp": 26.5,
    "atemp": 0.0,
    "D": 10.0,
    "d": -0.9830777902388692,
    "W": 9.01692220976113,
    "leeway": 0.0,
    "cmg": 229,
    "tws": 21.4,
    "twa": -139,
    "twd": 89,
    "cdr": 149,
    "csp": 0.29,
    "prmsl": 0.0,
    "hum": 0.0
}

The transformer reads the data from the cache and generates such an object. Then it is sent to a WebSocket server.

To run this transformer example

Start the websocket server, on a port of your choice:

 $> node wsnmea.js -port:1234

Define your transformer in the properties file:

forward.07.type=wsp
forward.07.wsuri=ws://localhost:1234/

Make sure you have configured the Pebble application as required (WebSocket URI), and you are good to go.

Application list
Start here
Press select to start
Choose the channel
Scroll...
Channel list
Choose...
Hit select
Displayed!
Display

WebSockets

WebSocket protocol is supported, in input, and in output. If needed, you can start your own local WebSocket server, running on nodejs. To install it (once):

 $> npm install

Then, to run it,

 $> node wsnmea.js

or

 $> npm start
An alternative, Node-RED.

There is a possibility to use Node-RED as a forwarder as well, that is a cool one. You can create a Node-RED flow that ingests data from a TCP port, and spits them out on a WebSocket one.

Node-RED

Here is the flow to import in Node-RED.

To run it
  • Start the Mux ($> ./mux.sh)
    • Make sure there is a TCP Forwarder (port 7002)
  • Start the node script wsnmea.parser.js.
 node wsnmea.parser.js
  • Start Node-RED and run the flow mentioned above.

Node-RED

  • The Multiplexer reads the NMEA data and forwards them on a TCP Channel.
  • Node-RED listens to the TCP port, and forwards the payloads to a WebSocket server
  • The WebSocket server transforms the data it receives into the expected format, and broadcasts them to all the connected cliengs (Pebbles in this case).

Once everything runs, you can reach http://localhost:9876/data/web/wsconsole.html to see the data as they come through.

REST Admin Interface

The properties files like nmea.mux.proeprties defines the configuration at startup.

You can remotely manage the input channels and the re-broadcasting ones through a REST interface. The soft includes a dedicated HTTP Server. The http port is driven by a property (in nmea.mux.properties). Same if you want the HTTP server to be started or not.

with.http.server=yes
http.port=9999

This HTTP Server is designed and written to run on small computers (like the Raspberry PI Zero). It is NOT an enterprise server, and it will NOT scale as one.

Supported REST end-points

List of operations

A full list of the available REST services is available at

 GET /oplist

Operations List

Examples

All the end points and operations are defined in nmea.mux.GenericNMEAMultiplexer.java. See the List<Operation> named operations.

 GET /mux/serial-ports

returns a payload as:

[
  "/dev/tty.Bluetooth-Incoming-Port",
  "/dev/cu.Bluetooth-Incoming-Port"
]

The list of available serial ports.

 GET /mux/channels

returns a payload like

[
  {
    "cls": "consumers.client.SerialClient",
    "type": "serial",
    "port": "/dev/ttyUSB0",
    "br": 4800
  },
  {
    "cls": "consumers.client.BME280Client",
    "type": "bme280"
  }
]
 GET /mux/forwarders

returns a payload like

[
  {
    "cls": "forwarders.TCPWriter",
    "port": 7001,
    "type": "tcp"
  },
  {
    "cls": "forwarders.ConsoleWriter",
    "type": "console"
  }
]
 DELETE /mux/forwarders/:type

type is one of

  • file. requires a body like
{
   "log": "./data.nmea",
   "type": "file"
}

identical to the elements returned by GET /mux/forwarders.

  • console. requires no body.
  • tcp. requires a body like
{
     "port": 7002,
     "type": "tcp"
}

identical to the elements returned by GET /mux/forwarders.

  • ws. requires a body like
{
   "wsUri": "ws://localhost:9876/",
   "type": "ws"
}

identical to the elements returned by GET /mux/forwarders.

 DELETE /mux/channels/:type
 POST /mux/forwarders

with payloads like:

  • file. requires a body like
{
   "log": "./data.nmea",
   "type": "file"
}

identical to the elements returned by GET /mux/forwarders.

  • console. requires a body like
{
     "type": "console"
}
  • tcp. requires a body like
{
     "port": 7002,
     "type": "tcp"
}

identical to the elements returned by GET /mux/forwarders.

  • ws. requires a body like
{
   "wsUri": "ws://localhost:9876/",
   "type": "ws"
}

identical to the elements returned by GET /mux/forwarders.

 POST /mux/channels

There is a Web UI using the REST resources above.

Note: This Web UI is to be considered as an example of the way to access the resources. Nothing more. Best case scenario, a UI Developer might find it cute...

On the HTTP Port, use a url like http://machine-name:9999/web/admin.html, where machine-name is the name of the machine where the multiplexer is running, and 9999 is the port defined in the properties.

Admin Web UI

And any REST client (NodeJS, Postman, curl, your own code, ...) does the job.

Example with curl:

$ curl -v http://192.168.1.181:9999/mux/last-sentence
  *   Trying 192.168.1.181...
  * Connected to 192.168.1.181 (192.168.1.181) port 9999 (#0)
  > GET /mux/last-sentence HTTP/1.1
  > Host: 192.168.1.181:9999
  > User-Agent: curl/7.43.0
  > Accept: */*
  >
  < HTTP/1.1 200
  < Access-Control-Allow-Origin: *
  < Content-Length: 110
  < Content-Type: application/json
  <
  * Connection #0 to host 192.168.1.181 left intact
  {
      "last-data": "$GPGGA,171812.255,3718.7160,N,12142.3005,W,1,09,0.9,494.3,M,,,,0000*16",
      "timestamp": 1499466116699
  }
 $

With REST traffic

You can also visualize the REST traffic if needed; In (server to client) on the left, Out (client to server) on the right.

With graph

Click on the Flow value (top-right) to show/hide the graph.

Note: Don't be alarmed by the colors of the graph, they all rely on their own css class, named .graphdisplay.

The cool thing in this context is that whatever complex the rendering (what the user sees) looks like, the complexity belongs to the renderer (i.e. the client, the web browser in this case). The server is only responsible for providing the raw data the graph is based on; and typically here, providing the data is certainly not the most complex part, pinging the server does the job... As a matter of fact, a small board like a Raspberry PI Zero can totally assume it, without noticing.

Dynamic loading

You have the possibility to dynamically load Channels, Forwarders and Computers.

To load a class, mention its name in the properties file used at startup, like for example:

#
# Dynamic loading sample
#
forward.03.cls=nmea.forwarders.ProcessorSkeleton
#

This ProcessorSkeleton is part of the project, it is provided as an example you can start from to develop your own forwarders.

Note: Dynamically loaded classes can also be managed from the REST admin interface. Use the Custom label when creating them.

They will show up in the admin interface, with their id in italic. Admin Web UI Notice the line that says skeleton.

Among the forwarders available in this project some are to be dynamically loaded.

  • SSD1306Processor
  • MQTTPublisher (a cool one, for IoT)
  • Character Console
  • etc... list keeps growing.

Character Console

Warning: for now, you cannot load more than one dynamically loaded forwarder, consumer or computer of the same class. They are identified by their class name, you cannot have more than one object with a given class name in the lists. For example, if you have an MQTTPublisher that uses a broker URL like tcp://192.168.1.136:1883, you might very well want to have another one, using a broker URL like tcp://io.adafruit.com:1883. Currently, this is not possible.

This should not be too much of a problem though, specially if you use all this on a boat, you probably do not need in Internet connection. A small LAN would do the job in this case.

Compatibility with other softwares

Whatever software understands NMEA is happy with all this. For example, OpenCPN:

OpenCPN

In the screenshot above:

  • The atmospheric pressure and air temperature come from a BME280 (~ $20)
  • The True Wind is calculated by a Computer (as well as the current, not displayed here, but available)
  • Heel and Pitch come from an LSM303 (~ $15)
  • Data are re-broadcasted on TCP

All the data can be logged, and replayed. Replayed data can be displayed just like above, in OpenCPN or any NMEA-savvy software. The screenshot above has been taken during one of those replays (that explains the disparity between air and water temperature...)

Some (good and open source) softwares

A note on the build

As you must have seen, we use Gradle here, along with the ShadowJar plugin. This plugin gathers all the dependencies, and puts them all together in a single jar file (named a fat-jar). This makes your life considerably easier, just put the jar named build/libs/NMEA.multiplexer-1.0-all.jar in your classpath (see for example in the mux.sh), and you have everything you need in there.

This fat-jar is currently (Apr-2017) a bit more than 7 Mb big. In some cases, you might want to trim the jar down, to make it smaller. If you do not need the classes that can be dynamically loaded, if you are not interested in the I2C sensors, etc, you could very well prune those projects, the resulting jar can be substantially smaller. To do so for yourself, refer to the Gradle documentation.

Open Issues

  • UDP client
    • Needs some cleanup...

Remote access

Remote access is not a problem on any Raspberry PI, ssh does the trick. It can even be done from phone or other small devices. For example, several ssh clients are available on Android, here is a screenshot from a phone:

SSH Console

In this kind of case, it could be a good idea to have scripts with small names... See go and killmux, for examples.

The Raspberry PI can also be an hotspot, you connect to its network to be able to access the data. there is a good tutorial for that on the Adafruit website. This one also, is worth a look.

The definition of your network is in /etc/hostapd/hostapd.conf.

scp and other such tools are also conveniently available.

This can prove to be particularly convenient when you want to log data away from a stable Internet connection (in your car, in a plane, on a boat, etc). You start the Raspberry PI, with a hotspot generating its own network. Then you can connect on that network, and you can drive the Multiplexer application (start, stop, monitor, etc).

Along the same lines - even if it can sound a bit disproportionate - VNC works just fine, even on the Raspberry PI Zero (see below)

VNC Viewer

This is Jessie/Pixel running on a Raspberry PI Zero.

Note: Access point and Internet access

To enable hostapd to have you Raspberry PI acting as a WiFi hotspot, as we said, you can follow those instructions from the Adafruit website.

The thing is that when the Raspberry PI becomes a WiFi hotspot, you cannot use it to access the Internet, cannot use apt-get install, cannot use git pull origin master, etc, that can rapidely become quite frustrating.

In the past, I had written a couple of scripts to juggle with the various configuration files (hostapd.conf, wpa_supplicant.conf, /etc/network/interfaces, etc). This worked for a while, then after an apt-get upgrade, it stopped working, some config files had changed... Bummer.

A much better approach seems to be to have 2 WiFi adpaters. The Rasberry PI 3 and the Zero W already have one embedded, I just added another one, the small USB WiFi dongle I used to use on the other Raspberry PIs. This one becomes named wlan1. All I had to do was to modify /etc/network/interfaces:

# interfaces(5) file used by ifup(8) and ifdown(8)

# Please note that this file is written to be used with dhcpcd
# For static IP, consult /etc/dhcpcd.conf and 'man dhcpcd.conf'

# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d

auto lo
iface lo inet loopback

iface eth0 inet manual

allow-hotplug wlan0
iface wlan0 inet static
  address 192.168.42.1
  netmask 255.255.255.0

#iface wlan0 inet manual
#    wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

allow-hotplug wlan1
iface wlan1 inet dhcp
wpa-ssid "ATT856"
wpa-psk "<your network passphrase>"

See the 4 lines at the bottom of the file, that's it!

Now, when the wlan1 is plugged in, this Raspberry PI is a WiFi hotspot, and has Internet access.

Start / Stop all forwarders

In some cases, you might want to start and stop the forwarders (doing logging for example) on demand. Like when you want to log data when driving, hiking, running, etc.

There is a possibility not to start the forwarders when the multiplexer is starting. For that, set the system variable process.on.start to false, default value is true.

In any case, the value can be set using the service

 PUT /mux/mux-process/on

or

 PUT /mux/mux-process/off

The current status can be read using

 GET /mux/mux-process

A Web UI (using the above) is also available:

  http://machine:9999/web/runner.html

Screenshots from a smart-phone:

Notice that this allows you to set the process.on.start, but also to shutdown the Multiplexer. Use it with caution.

Here is a use case:

You start from your home for a run, you want to log the data from a GPS. You start everything at home before you go, with process.on.start=false. All the stuff (Raspberry Pi Zero W, battery, GPS) is in a plastic box in your backpack, a Nalgene canteen, or something similar. You get out, and when you're about to start running, you connect to the Raspberry PI from your cell phone, and from the UI above, you start the recording. Once done, you stop it the same way. The data have been collected only when you were running. Then you can even stop the Multiplexer, before going home to download and visualize your data file.

Demos

Follow this link.

Examples!!

They are here.

Also see a combination with the SunFlower project. This takes the position from a GPS (serial port), and the device heading from an LSM303. Then it orients a solar panel so it faces the sun. The specific forwarder that does the job is the GPS.sun.servo project.

And there is an integration with the REST Nav Server.

Case Studies

Bonus

  • A sample of a log file analyzer is in util.LogAnalyzer:
 $ java -cp ./build/libs/NMEA.multiplexer-1.0-all.jar util.LogAnalyzer ./sample.data/2017.06.17.nmea

  Started 17-Jun-2017 09:11:01 PDT
  Arrived 17-Jun-2017 11:42:37 PDT
  9,097 record(s) out of 36,387. Total distance: 12.221 km, in 2 hour(s) 31 minute(s) 36.0 sec(s). Avg speed:4.837 km/h
  Max Speed: 6.112 km/h
  Top-Left    :N  37°20.17' / W 121°43.62'
  Bottom-Right:N  37°18.70' / W 121°41.46'

 $
  • A sample of a kml file generator is in util.NMEAtoKML:
 $ java -cp ./build/libs/NMEA.multiplexer-1.0-all.jar util.NMEAtoKML sample.data/alcatraz.2018.may.5.nmea --title:Alcatraz --sub-title:"May 5, 2018"
Started 05-May-2018 10:32:23 PDT
Arrived 05-May-2018 15:52:31 PDT
38,426 record(s) out of 76,821. Total distance: 44.222 km, in 5 hour(s) 20 minute(s) 8.0 sec(s). Avg speed:8.288 km/h
Max Speed: 327.804 km/h
Min alt: -248.60 m, Max alt: 212.00 m, delta 460.60 m
Top-Left    :N  37°49.84' / W 122°30.66'
Bottom-Right:N  37°44.91' / W 122°24.12'

Generated file sample.data/alcatraz.2018.may.5.nmea.kml is ready.
 $

Then open the generated file in Google Earth.