Problem With The Backup: collecting data to Node

[mz27ethio]

Greetings all,
As you can see in the attached images, I have encountered problem related with issue number #105 #127 and #149 . The problem started after I updated the node. Description of the problem:

• on the main page for the node web UI, "time since backup" is shown as N/A (Picture shown)
• Also, after the start of tracking, it shows that the file will be saved as /None (Picture shown)
• Automatic backup is not working.

The version of the node:

As it was not allowing automatic backup, I tried forcing it as shown here and this was the result:

`cd /opt/ethoscope-node/node_src/scripts

sudo python backup_tool.py -e 1

Forcing backup for ethoscope 001
INFO:root:Initiating backup for device 001d54d8d1c94e43979d5805c9d995a7
INFO:root:Running backup for device 001d54d8d1c94e43979d5805c9d995a7
INFO:mysql.connector.authentication:package: mysql.connector.plugins
INFO:mysql.connector.authentication:plugin_name: mysql_native_password
INFO:mysql.connector.authentication:AUTHENTICATION_PLUGIN_CLASS: MySQLNativePasswordAuthPlugin
INFO:root:Initializing local database static tables at /ethoscope_data/results/None
INFO:root:Making parent directories
WARNING:root:[Errno 17] File exists: '/ethoscope_data/results'
INFO:root:Ensuring DAM file exists at /ethoscope_data/results/None.txt
INFO:root:Database mirroring initialised
ERROR:root:Cannot mirror the 'SENSORS' table
ERROR:root:1146 (42S02): Table 'ETHOSCOPE_001_db.SENSORS' doesn't exist
INFO:root:Backup status for 001d54d8d1c94e43979d5805c9d995a7 is 100.00%
INFO:root:Backup done for for device 001d54d8d1c94e43979d5805c9d995a7 `

Then when I checked the /ethoscope_data/results directory, it has saved the data as NONE.db and NONE.txt

Also, I tried checking the status of the ethoscope_backup.service in systemctl and this was the result:

`● ethoscope_backup.service - Ethoscope node server
Loaded: loaded (/usr/lib/systemd/system/ethoscope_backup.service; enabled; preset: disabled)
Active: active (running) since Wed 2023-02-08 18:10:57 CST; 5min ago
Main PID: 429 (python)
Tasks: 1 (limit: 18983)
Memory: 35.3M
CPU: 231ms
CGroup: /system.slice/ethoscope_backup.service
└─429 /usr/bin/python /opt/ethoscope-node/node_src/scripts/backup_tool.py

Feb 08 18:11:03 node python[429]: urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
Feb 08 18:11:03 node python[429]: During handling of the above exception, another exception occurred:
Feb 08 18:11:03 node python[429]: Traceback (most recent call last):
Feb 08 18:11:03 node python[429]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
Feb 08 18:11:03 node python[429]: self.run()
Feb 08 18:11:03 node python[429]: File "/opt/ethoscope-node/node_src/ethoscope_node/utils/backups_helpers.py", line 148, in run
Feb 08 18:11:03 node python[429]: active_devices = self.find_devices()
Feb 08 18:11:03 node python[429]: File "/opt/ethoscope-node/node_src/ethoscope_node/utils/backups_helpers.py", line 89, in find_devices
Feb 08 18:11:03 node python[429]: logging.error("The node ethoscope server %s is not running or cannot be reached. A list of available ethoscopes could not be found." % server)
Feb 08 18:11:03 node python[429]: NameError: name 'server' is not defined`

As I am a beginner, I wanted to ask if you guys can help me on how to resolve this problem. I have several experiments lined up, so urgent help is much appreciated.

Thanks

[ggilestro]

Hi Michael - this should all be solved with early commits. Are you running the latest version of the software on the ethoscope too?

[mz27ethio]

Hello Giorgio,

Thanks for your response. I tried updating the Ethoscope through the web interface (http://node:8888)

However, it is not working. The Ethoscope version is the same before and after the update.

So, I opted for downloading latest version of the Ethoscope (the January 30, 2023 release) from the Notion Page and tried using Balena Etcher to burn it.

However, I encountered two problems:

1. When I loaded the downloaded file (img.tar.gz) to the Balena Etcher software, it throws the following warning message:

2. After the burning process is completed, I inserted the SD into the Ethoscope, however, the Ethoscope is not coming online.

Is the latest img viable or is it a problem on my side? If so, please let me know how I can go ahead and update the Ethoscope.

Thanks

[ggilestro]

I see. The previous image had a little bug that interfered with update. The current one has had an upload issue perhaps. I'll try to fix this with a new image in the next few days or else you can try burn the previous one again an I'll explain how to do a manual update. Sorry for the inconvenience

…

On Fri, 10 Feb 2023, 00:38 Michael Ewnetu, ***@***.***> wrote: Hello Giorgio, Thanks for your response. I tried updating the Ethoscope through the web interface (http://node:8888) [image: photo_5992234085734530854_y] <https://user-images.githubusercontent.com/91686165/217938664-406f3eb9-5170-4f47-8aab-7de9a385176c.jpg> However, it is not working. The Ethoscope version is the same before and after the update. [image: photo_5992234085734530855_y] <https://user-images.githubusercontent.com/91686165/217938749-500c7a39-9308-49e3-b459-c9bda3322423.jpg> So, I opted for downloading latest version of the Ethoscope (the January 30, 2023 release) from the Notion Page <https://www.notion.so/The-ethoscope-60952be38787404095aa99be37c42a27> and tried using Balena Etcher to burn it. However, I encountered two problems: 1. When I loaded the downloaded file (img.tar.gz) to the Balena Etcher software, it throws the following warning message: [image: photo_5994485885548215082_y] <https://user-images.githubusercontent.com/91686165/217970248-2ec2909d-1eb2-44fc-abb5-12832d951932.jpg> 1. After the burning process is completed, I inserted the SD into the Ethoscope, however, the Ethoscope is not coming online. Is the latest img viable or is it a problem on my side? If so, please let me know how I can go ahead and update the Ethoscope. Thanks — Reply to this email directly, view it on GitHub <#178 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFKERPXI4GSXRUBR6HC4QLWWWEYLANCNFSM6AAAAAAUV5GDLU> . You are receiving this because you commented.Message ID: ***@***.***>

[rjs360]

Dear Giorgio and team, after a long caveat I have taken up working with the ethoscope platform again, and decided to update both my node and my ethoscopes to the latest versions (via the Notion site as @mz27ethio indicated).

I did this today, so these are what I assume are the most up-to-date images.

I now have what looks like an almost identical issue. For me, automatic video backup is inactive, and when I attempt to move the slider in the GUI's management settings, it reverts to inactive again after a while. At any rate, no video backups are generated during a tracking bout, nor during a video recording. My .txt and .db recording files are also labeled as "none.txt" and "none.db", but the "none.txt" file does seem to contain count data. I can provide more detailed information tomorrow when I am back in the lab, if needed.

Did this issue have a solution in the end?

Thanks.

[ggilestro]

Sorry, just to make sure we are on the same page: what is it that you are trying to do?
Backing up videos or backing up tracking data?

ethoscope_backup is the service that is copying tracking data (db format)
ethoscope_video_backup is the service copying video data, generated only when you use the "Video -> Record Video" button.

No video are generated by tracking.

[rjs360]

Dear Giorgio,
I tried doing both. Both seem to have a problem. I have attached below some images that may clarify the situation. When I connect to my ethoscopes this is the default "settings" state in the Management section:

I can move the slider for the video backup to activate it, but after a while or a refresh of the page, it will revert to inactive. I am also getting related error messages in the log (note that this is not the complete log):

I just tried again to record a video (no tracking), and while the interface indicated that a video was recording (I even got a live video window this time), and would be saved to the node, it did not record any video to the node (i.e. the video folder is empty).

I also tried a tracking session, for which the interface also indicated that a file would be saved to the node (with the typical, long filename), but it did not actually save any tracking data to the node once I stopped the tracking.

It does not seem to be a browser issue, as this happens the same way in Firefox and Midori (the default browser in my manjaro linus distro).

Does this help diagnose the problem?
-Ruud

[rjs360]

A small update on this issue:

I attempted to add the ethoscope_backup_service by running the set of commands indicated by @antortjim in issue #146:

"cp /opt/ethoscope-node/scripts/ethoscope_video_backup.service /etc/systemd/system
systemctl daemon-reload # refresh database of services
systemctl start ethoscope_video_backup # start the service
systemctl status ethoscope_video_backup # see if the service works"

but it seems this is not working:

I then tried to at least confirm that my devices (node and ethoscopes) are updated properly; here's an image indicating the status:

I am not sure what "software broken" means, but this can't be good?
Any suggestions would be most appreciated.
-Ruud

[ggilestro]

Software broken indicates some problem with the update procedure that is based on git.
The update server should show you something like the screenshot below. Not sure what is wrong there.

For the backup, I will need to see more logs to understand what is going on.
Running the following should give you some logs:
journalctl -u ethoscope_backup -rb

[antortjim]

As a workaround for now, you can download your videos from the ethoscope with scp or rsync.
With scp the command line is quite simple, this should work:

# from a terminal in the node
scp -ru alarm@ethoscope_ip:/ethoscope_data/results/ /folder/in/node/where/you/want/your/videos

You will get a bunch of .h264 files under /folder/in/node/where/you/want/your/videos/long_id/ETHOSCOPE_XXX/date_time which you can compile together into a single file with the script @ggilestro provides here

https://github.com/gilestrolab/ethoscope/blob/master/scripts/tools/ethoscope_videos2mp4.sh

which you run I believe like so

# requires ffmpeg installed
./ethoscope_videos2mp4.sh /folder/in/node/where/you/want/your/videos/long_id/ETHOSCOPE_XXX/date_time

Good luck!
A

[rjs360]

Hi Giorgio & Antonio,
Thanks for the support!
I had another look at the update server today and I am not getting the Software Broken message anymore; now it is showing this (while I am connected to my local ethoscope_wifi network):

I tried running another update while on my regular wifi (with internet), but that still doesn't work. I get a large error log that I have pasted below. I also ran the command that Giorgio suggested, and that output is listed at the end of this message:

Output by the GUI when attempting to update the node and ethoscopes via http://node:8888/:

"Traceback (most recent call last): File "/usr/lib/python3.10/urllib/request.py", line 1348, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "/usr/lib/python3.10/http/client.py", line 1282, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 975, in send self.connect() File "/usr/lib/python3.10/http/client.py", line 941, in connect self.sock = self._create_connection( File "/usr/lib/python3.10/socket.py", line 845, in create_connection raise err File "/usr/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/ethoscope_updater/update_server.py", line 178, in group response = updates_api_wrapper(device['ip'], device['id'], what='device/update') File "/opt/ethoscope-node/scripts/ethoscope_updater/helpers.py", line 300, in updates_api_wrapper raise e File "/opt/ethoscope-node/scripts/ethoscope_updater/helpers.py", line 286, in updates_api_wrapper f = urllib.request.urlopen(req) File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python3.10/urllib/request.py", line 519, in open response = self._open(req, data) File "/usr/lib/python3.10/urllib/request.py", line 536, in _open result = self._call_chain(self.handle_open, protocol, protocol + File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain result = func(*args) File "/usr/lib/python3.10/urllib/request.py", line 1377, in http_open return self.do_open(http.client.HTTPConnection, req) File "/usr/lib/python3.10/urllib/request.py", line 1351, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [Errno 113] No route to host>"

And here is the output of:
[rs@node ~]$ journalctl -u ethoscope_backup -rb

May 11 13:42:37 node python[506]: NameError: name 'server' is not defined
May 11 13:42:37 node python[506]: logging.error("The node ethoscope server >
May 11 13:42:37 node python[506]: File "/opt/ethoscope-node/node_src/ethoscop>
May 11 13:42:37 node python[506]: active_devices = self.find_devices()
May 11 13:42:37 node python[506]: File "/opt/ethoscope-node/node_src/ethoscop>
May 11 13:42:37 node python[506]: self.run()
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/threading.py", li>
May 11 13:42:37 node python[506]: Traceback (most recent call last):
May 11 13:42:37 node python[506]: During handling of the above exception, anoth>
May 11 13:42:37 node python[506]: urllib.error.URLError: <urlopen error [Errno >
May 11 13:42:37 node python[506]: raise URLError(err)
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py>
May 11 13:42:37 node python[506]: return self.do_open(http.client.HTTPConne>
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py>
May 11 13:42:37 node python[506]: result = func(*args)
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py>
May 11 13:42:37 node python[506]: result = self._call_chain(self.handle_ope>
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py>
May 11 13:42:37 node python[506]: response = self._open(req, data)
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py>
May 11 13:42:37 node python[506]: return opener.open(url, data, timeout)
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py>
May 11 13:42:37 node python[506]: f = urllib.request.urlopen(req, timeout=t>
May 11 13:42:37 node python[506]: File "/opt/ethoscope-node/node_src/ethoscop>
May 11 13:42:37 node python[506]: Traceback (most recent call last):
May 11 13:42:37 node python[506]: During handling of the above exception, anoth>
May 11 13:42:37 node python[506]: ConnectionRefusedError: [Errno 111] Connectio>
May 11 13:42:37 node python[506]: sock.connect(sa)
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/socket.py", line >
May 11 13:42:37 node python[506]: raise err
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/socket.py", line >
May 11 13:42:37 node python[506]: self.sock = self._create_connection(
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/http/client.py", >
May 11 13:42:37 node python[506]: self.connect()
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/http/client.py", >
May 11 13:42:37 node python[506]: self.send(msg)
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/http/client.py", >
May 11 13:42:37 node python[506]: self._send_output(message_body, encode_ch>
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/http/client.py", >
May 11 13:42:37 node python[506]: self.endheaders(body, encode_chunked=enco>
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/http/client.py", >
May 11 13:42:37 node python[506]: self._send_request(method, url, body, hea>
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/http/client.py", >
May 11 13:42:37 node python[506]: h.request(req.get_method(), req.selector,>
May 11 13:42:37 node python[506]: File "/usr/lib/python3.10/urllib/request.py>
May 11 13:42:37 node python[506]: Traceback (most recent call last):
May 11 13:42:37 node python[506]: Exception in thread Thread-1:
May 11 13:42:37 node python[506]: Hit Ctrl-C to quit.
May 11 13:42:37 node python[506]: Listening on http://0.0.0.0:82/
May 11 13:42:37 node python[506]: Bottle v0.12.23 server starting up (using WSG>
May 11 13:42:37 node python[506]: INFO:root:Starting backup round
May 11 13:42:36 node systemd[1]: Started Ethoscope node server.

Hope that helps narrow it down a bit.
-Ruud

[antortjim]

1. Make sure the Python code you are running in the node is up to date (run git pull in the folder in the pc that contains the node code)
2. Your problem is with ethoscope_backup or ethoscope_video_backup? I.e. are you having trouble getting dbfiles or the raw videos? You mentioned first ethoscope_video_backup but now show the output for ethoscope_backup. Does either of them work? If none of them work, you may have a problem with the WiFi network configurations.

[rjs360]

Dear Antonio,

Re 1. I will attempt this soon when I have access to my node, and report back.

Re 2. At this time, I am not receiving any .db files nor videos on the node after either tracking attempts, or video recordings. I tried both processes several times, and thought there was something wrong with either the node version or ethoscope version (or both), which is why I am attempting to update them (unsuccessfully). Giorgio asked me to run the "journalctl -u ethoscope_backup -rb" command for troubleshooting purposes, so I submitted the output of that.

I'll let you know what happens when I perform the git pull.
-Ruud

[rjs360]

ok, apologies if this is the wrong folder to run the "git pull" command in but this is the output for running "sudo git pull" in:
/opt/ethoscope-node/

"error: unable to normalize alternate object path: /tmp/ethoscope-node/ethoscope-node/objects"

It lists the above line 3 times and finishes with:
"Already up to date"

[rjs360]

Does anyone have any other suggestions/solutions?
Or do I need to provide more data?

Could I perhaps try some older combination of node/ethoscope images, to see if the latest upgrade caused the issue for me? I can only download the latest images at the moment.

Thanks for any help
-Ruud

[antortjim]

Dear @rjs360
Sorry for the late reply.

I am not sure what this means

"error: unable to normalize alternate object path: /tmp/ethoscope-node/ethoscope-node/objects"

but I think it's not important, since you are indeed getting finally the right message (already up to date).
The best wayt to make sure you have the updated code is running git rev-parse HEAD and checking the output matches the hash code of the last commit in https://github.com/gilestrolab/ethoscope (currently 666c6dae12d88d534fe3367c9ada965f2082b79c).

Please upload the logfiles created with these commands:

In the node

journalctl -u ethoscope_backup  --since "2 week ago" > db_backup.log
journalctl -u ethoscope_video_backup  --since "2 week ago" > video_backup.log

In the ethoscope, put this in a text file and run it. Let me know if you don't know how to do it

#! /bin/bash
journalctl -u ethoscope_device  --since "2 week ago" > db_backup.log
mysql -uethoscope -pethoscope -e "USE ETHOSCOPE_XXX_db; SHOW TABLES;" > mysql.log

for ROI in {1..20}; do
    echo ROI_${ROI} >> mysql.log
    mysql -uethoscope -pethoscope -e "USE ETHOSCOPE_XXX_db; SELECT COUNT(*) FROM ROI_${ROI};" >> mysql.log
done

where XXX is the 3 digit code of yoru ethoscope number :)

This info will help us debug your problem, with the information we have I think it's very hard.

Cheers,
A