Simple step-by-step howto for Linux (and Arduino Yun)

[drtrigon]

According to gioblu/PJON#211: I had already a look at ModuleInterface and it looks promising (e.g. the webserver/interface). But honestly I did not get how to set it up and what parts do what.

Basically I want to have a device (call it router) that connects to any number of PCs (honestly I do not care about strategy here) one one side and on the other a SoftwareBitBang to replace my 1-wire network. All can/should be local only - if I want to share then not the devices but the webserver/interface (which can sit on any local machine) only. So I am fine with any solution ThroughSerial, Ethernet, LUDP, ...

My current issue is that I am not able to establish a connection between PC and any device using any stragtegy, I already tried:

• Arduino Yun
• Arduino Uno
• also have Nanos, Micros, Megas, DUEs, ESPs lying around (but no ethernet shield)
• I tried; Ethernet, ThroughSerial, using PJON-python

As I have a linux machine (ubuntu 14.04) I really have trouble to compile PJON-gRPC and also ThroughSerial examples as they are for win only. Using the Yun is ok for me but out of my experience it always needs some special tricks thus I wanted to start simple and small (ThroughSerial). I imagine something like Arduino Uno connected via ThroughSerial to a linux machine. But I am fine with any starting point that works (minimal example, prove of concept) - this is my desperate need - can you help here?

I assume you are referring to ModuleInterface/examples/WebPage/WINDOWS_LINUX_ETCP and ModuleInterface/examples/WebPage/WINDOWS_LINUX_LUDP ? Can you give me a simple step-by-step howto on how to compile that for linux and arduino yun? How to debug in case it's not working?

[fredilarsen]

I agree that there are too many levels to build at once, so that it is difficult to get everything right at first try.

I have started on adding two new examples to help build the WebPage example setup gradually. When these are completed, I will include using them into the instructions.

Basically the steps will then be:

1. Build Arduino modules ARDUINO_SWBB\SensorMonitor and ARDUINO_SWBB\LightController.
2. For testing, build ARDUINO_SWBB\TestModuleMaster. This will connect to the same SWBB bus as the two worker modules, and send them configuration and do synchronization. Check that the LightController LED reacts to the light level of the SensorMonitor. If it does, the two worker modules are ready for use.
3. Disconnect the ARDUINO_SWBB\TestModuleMaster.
4. Build the BlinkingRGBSwitch PJON example with an Arduino with an Ethernet shield. Connect it to the SWBB bus and to your LAN with an Ethernet cable.
5. For testing, build WINDOWS_LINUX_LUDP\TestModuleMaster on a Linux PC connected to the same LAN. Run it and see that the light falling on the SensorMonitor influences the LightController. If it does, it means that you have tested worker modules and the connection through the LAN to the PC program.
6. Stop the WINDOWS_LINUX_LUDP\TestModuleMaster.
7. Install Apache and MySQL and load the database configuration and web page setup.
8. Build and run WINDOWS_LINUX_LUDP\ModuleMasterHttp and verify that the database shows live measurements from the SensorMonitor.
9. Check that the web page shows measurements and controls the modules. Turn light on and off by clicking the buttons.

Do you think this is a good plan to make it easier to get going?

[drtrigon]

@fredlarsen: I appreciate a more detailed docu - especially as I got the impression your code uses nice and clever tricks that I want to learn too... ;)

2. How many Arduinos are need at this point? 3? Does the TestModuleMaster give output via Arduino usual serial monitor?
3. Here we also need 3 Arduinos and 1 ETH shield, right?

Yes I think this will be a good starting point for a step-by-step (fool-prove ;) starting or install guide - at least I got it now. The thing is that PJON and all the possibilities can be confusing in the beginning, might be worth having something like a glossary page that explains and clarifies expressions (difference between surrogate and router, etc.). I do also think drawing, diagrams and schemes could help a lot, e.g.:

Setup 1 (step 1)
SensorMonitor -- SWBB -- LightController

Setup 2 (step 2)
SensorMonitor -- SWBB -- LightController
                  |
            TestModuleMaster (AVR)

Setup 3 (step 5)
SensorMonitor -- SWBB -- LightController
                  |
            BlinkingRGBSwitch
                  |
                 LUDP
                  |
            TestModuleMaster (PC)

Setup 4 (step 8)
SensorMonitor -- SWBB -- LightController
                  |
            BlinkingRGBSwitch
                  |
                 LUDP
                  |
            ModuleMasterHttp (PC/webserver)

(of course these are supposed to be drafts only - ugly but I hope it's clear)

[fredilarsen]

For your setup 2) 3 Arduinos will be needed, yes. Two workers and one master all connected on one SWB bus. In this simple setup the workers communicate through the master, and the master keeps some configuration that is sent to the workers when they connect. Directing a flashlight at the SensorMonitor's light sensor for a little while makes the LightController turn off its LED, showing communication. The master can also print some status text to the serial line (USB) for inspection of measured light level and current LED status etc.

For 3), adding an Ethernet shield and a LED (optional, for inspection) to the master and flashing it with the PJON BlinkingRGBSwitch example is enough. Then a master with the same simple functionality can be run on Linux or Windows, also printing status text in the console.

I think using Ethernet instead of Serial is nice because it gives more freedom to placement of equipment, it does not have to be close to a PC (or RPI).

I agree that diagrams for the different steps are essential, so I will include that.

Regarding the use of the master-slave principle, here are my recommended design principles (not illustrated very well by the above examples though):

• Modules should be autonomous as much as possible. For example, a temperature regulator module should continue regulating based on local sensor input if it loses connection to the bus or master. It can automatically persist the last setpoints it received from the master so that it even will continue working as before after a power outage. This is a much more robust design principle than having a sensor module and a heating module, and the regulation logic in another device like the master. If a module uses sensor inputs from other modules for advanced control, it could fall back to a simpler control if communication goes down.
• The master is the hub through which values are exhanged in a general way, between modules and to and from the database + web page. Normally no module-specific logic should be added to the master so that it can be kept general. Meta-logic, where outputs from multiple modules are analyzed together, could be added to a separate analysis module (a PC program?) and not by just adding code to the master.

Of course, these principles may not suit all purposes, but it is nice to be able to take down any part of the system without other modules getting problems. I will add an e-mail reporting module (PC program) to the repo in the not too distant future, giving notifications if values are outside limits or there are communication problems.

[drtrigon]

I think using Ethernet instead of Serial is nice because it gives more freedom to placement of equipment, it does not have to be close to a PC (or RPI).

I agree! For me Serial is more the beginners getting started setup as it can be run using any standard arduino setup (uno, nano, mega, etc.) without the need for any additional hardware. I think we should keep that in mind regarding the big arduino community around. IMHO; TS examples are therefore a must have.

Regarding the use of the master-slave principle, here are my recommended design principles (not illustrated very well by the above examples though):
Modules should be autonomous as much as possible. For example, a temperature regulator module should continue regulating based on local sensor input if it loses connection to the bus or master. It can automatically persist the last setpoints it received from the master so that it even will continue working as before after a power outage. This is a much more robust design principle than having a sensor module and a heating module, and the regulation logic in another device like the master. If a module uses sensor inputs from other modules for advanced control, it could fall back to a simpler control if communication goes down.
The master is the hub through which values are exhanged in a general way, between modules and to and from the database + web page. Normally no module-specific logic should be added to the master so that it can be kept general. Meta-logic, where outputs from multiple modules are analyzed together, could be added to a separate analysis module (a PC program?) and not by just adding code to the master.

I agree to those principles. As many devices as possible should run autonomously and also do direct comm not invoking the master at all. The master should anyway be as general and stupid as possible; as you said "The master is the hub through which values are exhanged in a general way" and it should only be used by modules/devices to exchange data if they are on separated SWBB buses. The PC program (server) should only communicate to the devices in order to configure them and get measurements that the meta-logic needs for analysing, displaying and storing. And may be to flashing/reprogramming devices/modules through the bus? (but that will be very advanced)

I stumbled over another question these days; How do you serialize your data? Do you use MessagePack or something similar?

[fredilarsen]

I have written the 2 new examples to make the building of the setup less of a monolithic task, and am testing and evaluating. I will push the results when completed, but it may take some days because of a full agenda

I too agree with what you write, but as it is implemented, nothing goes directly between modules, everything goes through the master. This was to avoid letting modules know about the location (bus/media, id) of other modules, so that they can be moved without modifying other modules, and also to reduce the collisions on the bus(es) and potentially exploit the bus better. The master taking initiative and modules replying will minimize collisions and retries, like with a relay pin being passed back and forth. (The "event" support is the exception to this, where a module can notify the master and potential listeners immediately if an observation is made, like movement or a button being pushed.)

The serialization is done according to a contract that is requested by the master from each module at startup. The contract has a checksum (contract id) that is passed along with each value telegram so that the name and data type of each variable do not have to be sent along with the value every time. This reduces the size of the value telegrams drastically. The master will request an updated contract if it receives values with a contract id not matching the contract it has received from that module previously.
A value telegram is binary and contains the contract id and the number of values included, plus a sequence of the bytes making up the values, in network byte order. The size of each value is known from the contract remembered by the master. So the value telegrams are quite compact, just a few bytes of overhead plus the sum of bytes for the values (1-4 bytes for each value).
A special case of the value telegrams is when the number of values included is less than in the contract. In this case, the contract position of each value is prefixed to the value. This is used when transferring only one or a few outputs as events, or when synchronizing a modified setting back from a module with some kind of user interface (I have some modules with LCD display and clickable rotary encoder).

[drtrigon]

I have written the 2 new examples to make the building of the setup less of a monolithic task, and am testing and evaluating.

Thanks a lot!

I will push the results when completed, but it may take some days because of a full agenda

No hurry - I have the exact same issue. ;)))

Master: I see - more complexity ... the master is not only for relaying but for collision "handling" and contracts - and the contracts already go into the serialize topic so the master is important for that also.

Serialization: I came across https://github.com/fredilarsen/ModuleInterface/blob/master/examples/SensorMonitor/SensorMonitor.ino#L31 which led me to https://github.com/fredilarsen/ModuleInterface/blob/master/src/MI_PJON/PJONModuleInterface.h#L14. As I understand this is what you explained in the last paragraph and you use PJONModuleInterface to serialize your data. In more detail the specific functions are given here https://github.com/fredilarsen/ModuleInterface/blob/master/src/MI/ModuleVariableSet.h#L450. Given the complexity of the whole protocol and what you described this seems to me way more elaborate than "just using MessagePack" as I imagined. So I will keep that in mind and try do use PJONModuleInterface for my project as well. Thanks again!

[fredilarsen]

I now added a ARDUINO_SWBB/TestModuleMaster example and more instructions to the WebPage example README. This should make it easy to verify the two Arduino worker modules themselves before building more on top.

Also added a WINDOWS_LINUX_LUDP/TestModuleMaster example. This makes it possible to verify that the SWBB modules can be controlled by a master process on a PC through the LAN and a LUDP-SWBB Switch. When this runs as expected, all the hardware and PJON+ModuleInterface communication is working and only the web page and database setup remains.

Feedback on these additions, and potential remaining difficult or unclear issues, are welcome.

[fredilarsen]

A related issue: The LUDP-SWBB switches in PJON used Ethernet.begin(mac) at startup to get an IP address via DHCP. If the DHCP server is temporarily unavailable at startup (may happen after a power failure because the DHCP server may use a long time to boot), it would fail to start properly.
I changed this so that it will repeat the DHCP request until success, and also maintain it afterwards. I sent a pull request to @gioblu so it should be available shortly from PJON master.

[fredilarsen]

Sorry, closed this issue by accident. Will leave it open until you have got things working.