Troubleshooting EM2
Table of contents generated with markdown-toc
All steps in the below troubleshooting guide are meant to be done in order. If implementing a step does not resolve the issue, undo that step and move on to the next step.
-
Unplug all sensors from the box. Ensure power cable, VGA cable and keyboard cable are correctly plugged into their correct ports. Try to turn box on.
-
Unplug power cable from box and ensure power cable is connected to boat's 12V DC power supply. Measure power supply voltage to ensure ~12DC is being provided.
-
Open up box and ensure box's internal battery is connected to box's PSU correctly.
-
Measure battery terminals to ensure ~12V is being provided by battery.
- If, while unplugged from the boat's 12VDC power supply, the box powers up under battery power, check the pins, cables and connections of the red and black wires connected from the box's "Power" port to the upper two terminals on the PSU's "power in" side. Ensure they are all seated correctly and connected to the proper pins/terminals. The top most terminal on the PSU's "power in" side is positive and the second from the top is ground.
- Plug box's power cable back in and measure voltage across PSU's positive out terminal and any ground, ensuring ~12V is leaving PSU. Be careful not to accidentally short out box. If voltage = 0 then check the PSU's fuse and replace as needed.
- Check connection of box's main power button. Ensure it's two wires connect to the two correct pins on the motherboard (its two pins should have red bases). Using a multi-meter test continuity by unplugging the wires from their pins and depressing the power button.
- Unplug motherboard's power cable and measure voltage across it. If its not getting ~12V DC then ensure its connection to PSU is good.
-
Ensure yellow wire from Power Relay to Pin Housing connected to box's main power button is connected correctly to pin#9 (lone pin).
-
Check continuity of any screw on motherboard to all 4 of the big bolts on bottom of box (not the ones attached to the feet). If continuity = 0 then visually inspect under motherboard to see if any of those bolts are touching and shorting out the motherboard. If they are, back off bolts until they are not touching and try powering on the box again. If box successfully powers on after this it is advised that control box be replaced as the motherboard may be permanently faulty.
-
Replace control box with known working control box.
The screenshot above shows a fully functional system. Notice all sensor boxes are green and double check all the information such as time/date and GPS location are correct and PSI and HD disk space are at an expected value.
Here's how to troubleshoot the various sensors and equipment that gets plugged into the control box.
- Inspect keyboard and its cable and ensure it is plugged into one of the two USB ports on the front of the box.
-
If neither USB ports work open control box and ensure that the short USB cables inside the box are connected from the two external box ports to any two of the motherboard's USB ports. You can try replacing those cables with known working cables.
-
Replace keyboard with known working keyboard.
-
Replace control box with known working control box.
-
Inspect monitor and its cables and ensure that both the power cable and VGA cable are correctly plugged into their respective ports on the outside of the control box.
-
Check that monitor's power light is on. The monitor is usually powered by the boat's AC power supply but a few boats do not have an on-board power supply. If this is the case, ensure that the monitor's power cable is connected properly to a 12V DC - 48V AC power inverter and that the inverter is wired correctly into the boat's 12V DC power supply.
-
If the monitor is getting power but not displaying anything there's two things we can try.
-
Replace the monitor's VGA cable with a known good VGA cable and ensure it's connection from the back of the monitor to the control box is good.
-
Open the control box up and check that the short internal VGA cable is correctly connected from the box's external VGA port to the motherboard's VGA port. If it is, try replacing the short VGA cable with a known good cable. NOTE Some of the newer EM2 boxes make use of a male-to-male VGA cable instead of the older male-to-female cables.
-
Replace monitor with a known good monitor.
-
Replace control box with a known good control box.
-
Visually inspect the RFID Scanner. Ensure that the RFID scanner is mounted against a wooden block and that if it is mounted with metal bolts, that those bolts are making no contact to anything else that is made of metal as they can create signal interference if they are.
-
Visually inspect the RFID Scanner's cable and ensure it is not damaged. It is likely that the cable will have been spliced during the installation. Ensure that the splice looks solid and secure.
-
Ensure that the RFID Scanner's cable is plugged into the correct port on the control box.
-
Unplug the RFID cable from the control box and, using a multi-meter, measure the voltage across its pin#1 and ground. Ensure it is getting ~12VDC from the control box.
-
Open the control box and ensure that all 4 of the RFID port's pins are seated correctly and in the correct positions (1 = Red/Power, 2 = Blue/Serial out, 3 = White/Serial in, 4 = Ground).
-
Ensure that the RFID's serial wires (blue and white) are plugged into the correct serial port (COM2) on the Motherboard and that the wires are connecting with COM2's pin#2 (white) and pin#3 (blue).
-
Inspect the raw data of the serial port (see below) to see if the issue is a GUI issue.
-
Plug in known working RFID scanner to box and try it. If it works, your RFID scanner needs to be replaced and reinstalled.
-
Replace control box with known good control box.
-
Visually inspect the GPS Antenna. If it is mounted inside the wheelhouse (vs. out on the deck/roof), ensure it is not mounted near a lot of metal or other cables as transmission leakage from them can interfere with its ability to correctly acquire its signal. Also ensure that it is able to "see" the sky (i.e, make sure its not mounted inside a closet).
-
Visually inspect the GPS cable and ensure it is not damaged. Replace if it is. Ensure it is plugged into the Antenna correctly and the other end is connected to the correct port on the control box.
-
Unplug the GPS cable from the control box and, using a multi-meter, measure the voltage across its pin#1 and ground. Ensure it is getting ~12VDC from the control box.
-
Ensure that the GPS cable's wires have been connected to the female MOLEX connector properly. Pin#1=Red, Pin#2=White, Pin#3=Gray, Pin#4=Orange,Purple,Red/White,Black. Ensure that the Orange/White wire is terminated and taped off to prevent shorting. ADD PHOTO
-
Open the control box and ensure that all 4 of the GPS port's pins are seated correctly and in the correct positions (1 = Red/Power, 2 = Blue/Serial out, 3 = White/Serial in, 4 = Ground).
-
Ensure that the GPS's serial wires (blue and white) are plugged into the correct serial port (COM1) on the Motherboard and that the wires are connecting with COM1's pin#2 (white) and pin#3 (blue).
-
Inspect the raw data of the serial port (see below) to see if the issue is a GUI issue.
-
Replace the GPS antenna with a known good GPS antenna and try it. If it works, your original GPS antenna is faulty. If it still doesn't work, replace the GPS cable as well and try it.
-
Replace control box with known good control box.
-
Visually inspect the PSI Sensor if able. This should be easy to do if it's mounted inside the ship somewhere (likely the engine room). If it is mounted outside near the hauler, it may be sealed in wax-cloth or something similar and you won't be able to access the sensor. Check the sensor for rust/corrosion. Check the sensor's cable and its splice. Ensure the splice looks connected, secure and waterproof. You may need to re-splice the cable if not.
-
Ensure the PSI cable is plugged into the correct port on the control box.
-
Unplug the PSI cable from the control box and, using a multi-meter, measure the voltage across its pin#1 and ground. Ensure it is getting ~12VDC from the control box.
-
Ensure that the PSI cable's wires have been connected to the female MOLEX connector properly. Pin#1=Red/Power, Pin#3=White/Signal, Pin#4=Black/Ground. ADD PHOTO
-
Open the control box and ensure that all 3 of the PSI port's pins are seated correctly and in the correct positions (1 = Red/Power, 3 = White/Signal in, 4 = Ground).
-
Ensure that the PSI's signal wire (white) is connected to the box's resistor board correctly (Position#2 beside the red wire in the 4-pin housing).
-
Ensure that the two other white wires in the 4-pin housing on the resistor board are secure and connected to the Arduino correctly (Position 3 of the resistor board to A0 pin on Arduino. Position 4 of the resistor board goes to A1 pin on the Arduino).
-
Ensure that the Arduino is connected to the motherboard properly via its micro-USB to USB cable. You should see a flashing LED on the Arduino if it is.
-
Inspect the raw data output of the Arduino to see if the issue is a GUI issue.
-
Ensure Arduino has been properly flashed. Run "Flashard 5V" in the OS.
-
Replace the control box with a known good control box.
-
Pull apart the splice in the PSI cable and re-solder it correctly.
-
Replace the PSI sensor with a known good PSI sensor.
-
Visually inspect the horn. Replace if damaged.
-
Visually inspect the horn's cable. Ensure the splice is secure and solid.
-
Unplug the horn from the control box and replace with a known working horn.
-
Open the control box and measure continuity between the Horn's Pin#4 (Ground) and the box's ground. Continuity should = 0. If not, check the horn's ground cable and pin to make sure it is connected from pin#4 of the horn port to the negative terminal block.
-
Ensure that the Horn's signal cable (blue) and pin are properly connected from pin#3 in the Horn port to the control box's amp.
-
Ensure that the Arduino is connected to the motherboard properly via its micro-USB to USB cable. You should see a flashing LED on the Arduino if it is.
-
Ensure that the amp's power wire (red) is properly connnect to the box's positive terminal block and that the amp's signal-in wire (white) is properly connected to the 11 pin on the Arduino.
-
Replace amp with a known working amp.
-
Ensure Arduino has been properly flashed. Run "Flashard 5V" in the OS.
-
Pull apart the splice in the horn cable and re-solder it correctly.
-
Replace control box with known good control box.
-
Visually inspect the camera for noticable damage. You may need to replace the camera. You should also check the camera's seals and look inside the camera to see if there is water damage.
-
Visually inspect the camera's ethernet cable for damage and/or to ensure it is connected properly to the camera and the POE injector (or POE switch if there are more than one camera). You may need to replace the cable.
-
Open the camera and look to see if you can see the flashing LEDs that will tell you whether or not the camera is getting power. If there is no power, unplug both ends of the Ethernet cable and test the cable with an Ethernet cable tester. Rewire the cable if needed plug it back in. Does the camera work now?
-
Ensure that the camera's ethernet cable is plugged into the POE injector (or POE switch) correctly and that the POE injector/switch ethernet cable is plugged into the "CAMERA" ethernet port on the control box.
-
Ensure that the POE injector/switch's power cable is connected correctly the the CAM PWR port on the control box. Check that the wires of that power cable are also connected correctly to the injector/switch on the other end. NOTE Many problems with the camera are often attributed to a loose connection between the POE Injector and the CAM PWR cables. The connection is rather weak and it's a problem that EcoTrust Canada is currently aware of and working on solving appropriately.
-
Open the control box and ensure that the short Ethernet cable is properly connected from the box's Ethernet port to the Ethernet port on the motherboard. Visually check to see if the Motherboard's Ethernet port LEDs are blinking. If not, power is not getting to the camera and the likely culprit is the POE Injector/Switch or the CAM PWR port.
-
Inspect the CAM PWR port. Unplug the power cable from it and measure the voltage across Pin#1 (Power) and ground. Ensure it is 12V DC. NOTE While the majority of the EM systems are capable of working for a time at less than 12V DC, the camera is the exception. It will not work with anything shy of 12V DC. Therefore, check to make sure the control box is getting 12V DC in from the boat's 12V DC power supply.
-
Check to make sure the em.conf file is configured correctly to work with the camera(s) (see below).
-
Check to make sure the camera(s) are configured correctly as per the instructions provided by their manufacturer (see below).
-
Replace the camera with a known good camera.
-
Replace the control box with a good camera.
-
If the data drive is red in the GUI, first try re-seating the drive correctly in the data drive enclosure.
-
Try replacing the data drive with a known good drive.
-
Open the control box and ensure that the Data Drive Enclosure's SATA power cable and SATA data cable are seated correctly in their correct ports on the drive. Also ensure the other ends of those cables are seated correctly in their ports on the motherboard.
-
Replace the data drive's SATA power cable and SATA data cable with known good cables.
-
Ensure the data drive enclosure's power cable is drawing 12V DC from the motherboard by measuring the voltage across it and the ground. It may be hard to find the power pin inside the cable. To do so, gently rub your multi-meter's lead across all the pins until you find it.
-
Replace the data drive enclosure with a known good enclosure.
-
Replace the control box with a known good box.
#EXTRA STUFF
##Local Networking To set up local networking with other computers outside the camera network, you need to copy the following file to the /var/ directory, so you have a file like "/var/network" if you want to ssh into the box.
https://github.com/Ecotrust-Canada/em-control-box/blob/maint/templates/network.template
-
When measuring Voltages inside the box, it is usually best to unplug the battery's cables. You can then plug the multi-meter's ground lead directly into the battery's ground cable which will leave your hands free to more precisely measure the voltage across any other part of the box without having to worry about accidentally shorting something. (ADD PHOTO)
-
Visually check to make sure all wires inside the control box are properly crimped (EXAMPLE PHOTOS).
One EM2 and up, use the screen command.
For the GPS
screen /dev/ttyS0
For the RFID
screen /dev/ttyS1
For the Arduino
screen /dev/ttyUSB0
Ensure VGA cable is fixed securely at back of control box and screen. If no change there is also a short VGA cable internally within the control box which should be checked. Test if it is the monitor at fault by either connecting a different monitor to the control box, or trying a different connection into the monitor.
Ensure the connection into the control box is good and the wires are correctly inserted into the connector plug. If possible test the voltage being produced by the sensor into the box using a multi-meter. Testing between the sensor cable to a ground should produce a reading of around 1v (between 0.8 -5v) if working correctly. Check over wire to see if any joins or kinks may have broken the connection.
(EM1) - In the terminal, typing sudo putty will bring up a Putty window. Select serial, and ttyUSB0 and hit okay. Here you will see P### -newline- B### -newline- and so on. the number following P represents the voltage to pin A0 on the Arduino. The number compared to 1023 will be equal to the ratio of the voltage received to the Arduino voltage. For example, if the Arduino is the 5 volt version, an idle reading of P100 is healthy for a 5VD configuration, because a healthy sensor will produce ~ 1 volt, which is then divided in two (.5V) inside the box by the voltage divider. The 5V Arduino will then read the .5V --> (.5 / 5) x 1023 ~ 100. If you are getting a low reading here but it is consistent, you can try lowering psi_vmin in the config file, but take note this may indicate a sensor change impending; the sensor seems to output lower voltage near the end of its life.
If the output format for /dev/ttyUSB0 is not: P### B### .... but instead:
P###:B###:P###:B###...
The Arduino is flashed with the code for EM2. If you're on an EM1 box, you'll need to have spare Arduinos with the correct program. If you're on an EM2 box and have the first format (newlines) you can access the Arduino code through /opt/em/arduino/rec-adc.pde, adjust the printing commands slightly, and flash the arduino with em flashard
Note that when wire to A0 is taken off sometimes value read from A1 "leaks" on to A0 (pin has floating voltage).
Note that in using this method one needs to know the config i.e: 3V, 3VD, ... etc
A low putty reading (or equivalently, voltmeter reading inside the box) with a known good sensor attached indicates wiring problems. if you have a sensor attached to an end that you can quickly pop on you should see this putty reading go back up to a healthy value. If not, this indicates a problem between the box terminals and the Arduino itself.
If a known good sensor produces a good signal but the current one is producing a low/unreliable signal you need to either change the line or the pressure sensor or both. The current strategy has been to make the box end on a long wire, and while the sensor end of the cable is still skinny and unattached to anything, wire it through the cabin as you would on an install. Cut the existing sensor cable (POWER OFF) and try the old sensor with the new cable; if it indeed was the sensor, the price of the cable is negligible compared to the cord, and it will be good to have a new cord anyways. You can solder or twist the wires together, noting whether the previous technician has used the convention that black is ground, or the uninsulated "plain" is ground, and see if that betters your connection. Still nothing? --> New sensor. In this case replace the sensor but make sure to put all of the shrink wraps you will ever use in this job on the unsoldered wires first.
By the way, you can check if 2/3 of the cable to the sensor is good after you've cut. if you go back and turn the power on to the sensor, you can measure near the sensor if the cable was at least delivering power.
(EM2) - Some differences are that instead of putty we can access this data through cat < /dev/ttyUSB0.
Also, pins should not float at weird voltages in EM2 when not hooked up as we set pins A0, A1, A2 to LOW.
Sometimes a box may contain a 3.3V arduino that has the pressure sensor wired directly into one of its analog pins. This is characterized by the voltage reading reaching a certain threshold and then dropping down to 0 (presumably as voltage exceeds the pin's limit. Then, as the voltage comes down into the acceptable range the voltage reading will jump back up to a legitimate/high value and then track back down with the falling pressure. This results in double peaks. You may need to add a voltage divider or replace the arduino with a 5V version. Important: you must use a corresponding 5V FTDI chip as well, or the problem won't go away.
A weak or distorted amp sound may be the result of a blown transistor in the horn amp (for boxes with internal amps). Check to see that with no input the voltage across the third stage (PN2222's) is 12V. If there is significantly less than this then one or more of those transistors is blown. This will usually be accompanied with a red-hot transistor. If both blow, you will begin to hear distorted amp sounds as the x-istors heat up. Replace the blown transistors and make sure the input line is clean of any errant signals.
- Check the wires are correctly inserted into the connector plug.
- Ensure the connection to the GPS (cable) is full secured at both ends. It may have come loose on the GPS end if the locking sleeve wasn't fully engaged.
- Try connecting a different GPS if possible to test before re-wiring or replacing the unit.
- Record the message in the red GPS sensor area on the screen. ie, does it say "NO GPS FIX" or "NO DATA" ?
- Ensure the connection to the GPS (cable) is fully secured at both ends. It may have come loose on the GPS end if the locking sleeve wasn't fully engaged.
- Restart the system.
- If none of these succeed, remove the gps mounting bracket, and put the GPS in plain view of the sky via a window for testing. Does this fix it? If so, or if it was just periodically cutting out at sea, it needs a new mounting location with no steel or other dense material blocking the line to the sky. This may be a downside of mounting the GPS units inside the wheelhouse and on some vessels we may find the need to move the aerial.
Test with more than one chip to ensure the problem is with the scanner. Ensure the connection into the control box is good and the wires are correctly inserted into the connector plug. Check over wire to see if any joins or kinks may have broken the connection. If the scanner has been used for many years ensure it has not worn to the aerial or chip. Check to see how scanner was mounted. RFID scanners should be mounted to a wooden block which in turn is mounted to the deck rail of the boat. If a scanner is mounted with metal bolts directly to a metal deck rail, signal interference may occur causing the RFID scanner to not perform properly. To test this, un-mount scanner and try scanning a known good RFID chip. If the scanner works, re-mount the scanner properly.
Sometimes the scanner looks as if it's not scanning but the scans are being registered in the system, and just not showing on the display. If the scanner is a translucent scanner, scan a few chips and look inside it for its green LED to blink. Either way, look inside /mnt/data/RFID_DATA.CSV and try to find your recent scans. If in fact they are there but not counting on the user display, know that this problem has been solved more than once by re-imaging the system. You can also open Putty (EM1 Only) with sudo putty, selecting ttys1 and scanning to see your scan pop up in that terminal as some characters.
Ensure the horn is plugged in and its cabling is undamaged. Ensure the amp is wired correctly inside the control box. Ensure the amp has power, on most amps a red light should be visible. Adjust the volume control and other settings to max. Test the connections from the amp to the control box are plugged securely. Ensure the speaker wire is connected and shows no sign of corrosion. View the horn to make sure it is not damaged.
(EM 1) - Try to make the horn sound by going into Putty (See PSI section above) and again selecting serial device ttyUSB0. Type (it won't show) F500D1000E. 500 here could be anything from 500-6000 (frequency) and D can be 500-10000 (milliseconds). E means execute the horn sound with these parameters. If you don't hear the sound, or see some weird format in Putty (see PSI section above for format), you might suspect an Arduino problem, and throwing in a new one of these is a fast check (It must be a programmed one for EM1). Another quick check is taking a new horn, and hooking it up to the amp directly. Try the instruction again. If it worked this time, you know it is either just the horn, the horn cable, or both. If, however, the horn sounded not through the amp, you can try to carefully put the horn wires into the horn port on the box. Again try to sound it in putty. If you get a sound here, but not directly from the amp, you can suspect the amp as a source of the problem. When you try a new amp, actually just try hooking it up to the existing horn and box. The amp is the last piece to suspect but if it is the only piece with a problem this may be the fastest way to fix the horn problem.
An extra note on horns - When a horn problem exists solely in conjunction with a psi problem, the Arduino should be the first thing to come into suspect.
If the OS drive is full, that means the system's been running with no data drive.
When possible, it's best to avoid letting it get like that, because in some cases it might be hard to recover. Whenever the system reads "using OS disk" (or is red for any reason) it needs to be fixed because it can lead to further problems.
Before cleaning up the OS drive, install a data drive and ensure it's recording there.
The system has two data recording locations, /var/em/data is the version on the OS drive, and /mnt/data is the DATA drive.
If you don't care about the data, you can delete data in the folder /var/em/data/ ie)
rm -rf /var/em/data/*
Be careful to enter the above command exactly, as deviations of it could cause issues. If you need to keep the data from the OS disk, just move everything to the data drive. ie)
rsync -av /var/em/data/ /mnt/data
Camera video does not show up in UI.
- hit ctrl-alt-backspace to reset the UI.
- ensure the camera, switch and computer are powered on and connected. Their network ports should all be blinking wherever plugged in.
- ensure camera's connected. In the command line type:
ping 1.1.1.1
If it says the camera is "Unreachable", there's a networking issue (ie, not physically connected, bad cable, not powered on, etc.)
To reset the EM service: systemctl restart em-rec.service
If there are intermittent cases of one or more cameras missing in your online video which has freshly been uploaded, Here are steps to investigate, in order:
-
Was the data interrupted while uploading, or didn't finish? Check the boat drive if you still have it. To catch this, we'd need to check as soon as the data's initially checked.
-
On the vessel, ensure 4 video files (one for each camera) is recording on the box when you're there.
-
One cause if #2 is negative is that you have a duplicate camera IP address. It could happen if a camera was replaced recently for instance.
If there's still an issue after these steps, let's work on this on our weekly tech call.