Problem With The Backup: collecting data to Node #178
Comments
|
Hi Michael - this should all be solved with early commits. Are you running the latest version of the software on the ethoscope too? |
|
Hello Giorgio, Thanks for your response. I tried updating the Ethoscope through the web interface (http://node:8888)
However, it is not working. The Ethoscope version is the same before and after the update.
So, I opted for downloading latest version of the Ethoscope (the January 30, 2023 release) from the Notion Page and tried using Balena Etcher to burn it. However, I encountered two problems:
Is the latest img viable or is it a problem on my side? If so, please let me know how I can go ahead and update the Ethoscope. Thanks |
|
I see. The previous image had a little bug that interfered with update. The
current one has had an upload issue perhaps. I'll try to fix this with a
new image in the next few days or else you can try burn the previous one
again an I'll explain how to do a manual update. Sorry for the inconvenience
…On Fri, 10 Feb 2023, 00:38 Michael Ewnetu, ***@***.***> wrote:
Hello Giorgio,
Thanks for your response. I tried updating the Ethoscope through the web
interface (http://node:8888)
[image: photo_5992234085734530854_y]
<https://user-images.githubusercontent.com/91686165/217938664-406f3eb9-5170-4f47-8aab-7de9a385176c.jpg>
However, it is not working. The Ethoscope version is the same before and
after the update.
[image: photo_5992234085734530855_y]
<https://user-images.githubusercontent.com/91686165/217938749-500c7a39-9308-49e3-b459-c9bda3322423.jpg>
So, I opted for downloading latest version of the Ethoscope (the January
30, 2023 release) from the Notion Page
<https://www.notion.so/The-ethoscope-60952be38787404095aa99be37c42a27>
and tried using Balena Etcher to burn it.
However, I encountered two problems:
1. When I loaded the downloaded file (img.tar.gz) to the Balena Etcher
software, it throws the following warning message:
[image: photo_5994485885548215082_y]
<https://user-images.githubusercontent.com/91686165/217970248-2ec2909d-1eb2-44fc-abb5-12832d951932.jpg>
1. After the burning process is completed, I inserted the SD into the
Ethoscope, however, the Ethoscope is not coming online.
Is the latest img viable or is it a problem on my side? If so, please let
me know how I can go ahead and update the Ethoscope.
Thanks
—
Reply to this email directly, view it on GitHub
<#178 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAFKERPXI4GSXRUBR6HC4QLWWWEYLANCNFSM6AAAAAAUV5GDLU>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
Dear Giorgio and team, after a long caveat I have taken up working with the ethoscope platform again, and decided to update both my node and my ethoscopes to the latest versions (via the Notion site as @mz27ethio indicated). I did this today, so these are what I assume are the most up-to-date images. I now have what looks like an almost identical issue. For me, automatic video backup is inactive, and when I attempt to move the slider in the GUI's management settings, it reverts to inactive again after a while. At any rate, no video backups are generated during a tracking bout, nor during a video recording. My .txt and .db recording files are also labeled as "none.txt" and "none.db", but the "none.txt" file does seem to contain count data. I can provide more detailed information tomorrow when I am back in the lab, if needed. Did this issue have a solution in the end? Thanks. |
|
Sorry, just to make sure we are on the same page: what is it that you are trying to do?
No video are generated by tracking. |
|
Dear Giorgio, I can move the slider for the video backup to activate it, but after a while or a refresh of the page, it will revert to inactive. I am also getting related error messages in the log (note that this is not the complete log): I just tried again to record a video (no tracking), and while the interface indicated that a video was recording (I even got a live video window this time), and would be saved to the node, it did not record any video to the node (i.e. the video folder is empty). I also tried a tracking session, for which the interface also indicated that a file would be saved to the node (with the typical, long filename), but it did not actually save any tracking data to the node once I stopped the tracking. It does not seem to be a browser issue, as this happens the same way in Firefox and Midori (the default browser in my manjaro linus distro). Does this help diagnose the problem? |
|
A small update on this issue: I attempted to add the ethoscope_backup_service by running the set of commands indicated by @antortjim in issue #146: "cp /opt/ethoscope-node/scripts/ethoscope_video_backup.service /etc/systemd/system but it seems this is not working:
I then tried to at least confirm that my devices (node and ethoscopes) are updated properly; here's an image indicating the status:
I am not sure what "software broken" means, but this can't be good? |
|
Software broken indicates some problem with the update procedure that is based on
For the backup, I will need to see more logs to understand what is going on. |
|
As a workaround for now, you can download your videos from the ethoscope with scp or rsync. You will get a bunch of .h264 files under https://github.com/gilestrolab/ethoscope/blob/master/scripts/tools/ethoscope_videos2mp4.sh which you run I believe like so Good luck! |
|
Hi Giorgio & Antonio,
I tried running another update while on my regular wifi (with internet), but that still doesn't work. I get a large error log that I have pasted below. I also ran the command that Giorgio suggested, and that output is listed at the end of this message: Output by the GUI when attempting to update the node and ethoscopes via http://node:8888/: "Traceback (most recent call last): File "/usr/lib/python3.10/urllib/request.py", line 1348, in do_open h.request(req.get_method(), req.selector, req.data, headers, File "/usr/lib/python3.10/http/client.py", line 1282, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1328, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1277, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib/python3.10/http/client.py", line 1037, in _send_output self.send(msg) File "/usr/lib/python3.10/http/client.py", line 975, in send self.connect() File "/usr/lib/python3.10/http/client.py", line 941, in connect self.sock = self._create_connection( File "/usr/lib/python3.10/socket.py", line 845, in create_connection raise err File "/usr/lib/python3.10/socket.py", line 833, in create_connection sock.connect(sa) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/opt/ethoscope_updater/update_server.py", line 178, in group response = updates_api_wrapper(device['ip'], device['id'], what='device/update') File "/opt/ethoscope-node/scripts/ethoscope_updater/helpers.py", line 300, in updates_api_wrapper raise e File "/opt/ethoscope-node/scripts/ethoscope_updater/helpers.py", line 286, in updates_api_wrapper f = urllib.request.urlopen(req) File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python3.10/urllib/request.py", line 519, in open response = self._open(req, data) File "/usr/lib/python3.10/urllib/request.py", line 536, in _open result = self._call_chain(self.handle_open, protocol, protocol + File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain result = func(*args) File "/usr/lib/python3.10/urllib/request.py", line 1377, in http_open return self.do_open(http.client.HTTPConnection, req) File "/usr/lib/python3.10/urllib/request.py", line 1351, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [Errno 113] No route to host>" And here is the output of: May 11 13:42:37 node python[506]: NameError: name 'server' is not defined Hope that helps narrow it down a bit. |
|
|
Dear Antonio, Re 1. I will attempt this soon when I have access to my node, and report back. Re 2. At this time, I am not receiving any .db files nor videos on the node after either tracking attempts, or video recordings. I tried both processes several times, and thought there was something wrong with either the node version or ethoscope version (or both), which is why I am attempting to update them (unsuccessfully). Giorgio asked me to run the "journalctl -u ethoscope_backup -rb" command for troubleshooting purposes, so I submitted the output of that. I'll let you know what happens when I perform the git pull. |
|
ok, apologies if this is the wrong folder to run the "git pull" command in but this is the output for running "sudo git pull" in: "error: unable to normalize alternate object path: /tmp/ethoscope-node/ethoscope-node/objects" It lists the above line 3 times and finishes with: |
|
Does anyone have any other suggestions/solutions? Could I perhaps try some older combination of node/ethoscope images, to see if the latest upgrade caused the issue for me? I can only download the latest images at the moment. Thanks for any help |
|
Dear @rjs360 I am not sure what this means but I think it's not important, since you are indeed getting finally the right message (already up to date). Please upload the logfiles created with these commands: In the node In the ethoscope, put this in a text file and run it. Let me know if you don't know how to do it where This info will help us debug your problem, with the information we have I think it's very hard. Cheers, |
|
I can be more helpful if you allow me to access your system remotely, feel free to share the credentials if you want in a PM to antonio.ortega@kuleuven.be. Easiest way is via a remote desktop application like TeamViewer or AnyDesk |
|
Hi all, Some things that I see in the already sent logs:
|
|
Thank you Antonio!
I will try to provide the information you are looking for, but I am not
sure exactly how to get into the ethoscope to run a bash script. Do I need
to attach the ethoscope to a monitor/keyboard etc? Is that the easiest way?
…-Ruud
On Wed, May 24, 2023 at 3:13 PM Antonio ***@***.***> wrote:
Dear @rjs360 <https://github.com/rjs360>
Sorry for the late reply.
I am not sure what this means
"error: unable to normalize alternate object path: /tmp/ethoscope-node/ethoscope-node/objects"
but I think it's not important, since you are indeed getting finally the
right message (already up to date).
The best wayt to make sure you have the updated code is running git
rev-parse HEAD and checking the output matches the hash code of the last
commit in https://github.com/gilestrolab/ethoscope (currently
666c6da).
Please upload the logfiles created with these commands:
In the node
journalctl -u ethoscope_backup --since "2 week ago" > db_backup.log
journalctl -u ethoscope_video_backup --since "2 week ago" > video_backup.log
In the ethoscope, put this in a text file and run it. Let me know if you
don't know how to do it
#! /bin/bash
journalctl -u ethoscope_device --since "2 week ago" > db_backup.log
mysql -uethoscope -pethoscope -e "USE ETHOSCOPE_XXX_db; SHOW TABLES;" > mysql.log
for ROI in {1..20}; do
echo ROI_${ROI} >> mysql.log
mysql -uethoscope -pethoscope -e "USE ETHOSCOPE_XXX_db; SELECT COUNT(*) FROM ROI_${ROI};" >> mysql.log
done
where XXX is the 3 digit code of yoru ethoscope number :)
This info will help us debug your problem, with the information we have I
think it's very hard.
Cheers,
A
—
Reply to this email directly, view it on GitHub
<#178 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABSQSRR7OJAFXUIS5PH6QD3XHZMVXANCNFSM6AAAAAAUV5GDLU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
This error Seems to be linked to an unsuccessful git installation. |
|
@rjs360 yes, that's gonna be the easiest. You can use a terminal text editor like nano then you can run it like so Good luck |
|
Dear Antonio, Thanks and please let me know how I can help you determine what is going on exactly. db_backup.log |
|
Dear Ruud, |
|
Is it normal for the node to NOT have yay installed? I cannot find it with
whereis or which yay.
…-Ruud
On Thu, May 25, 2023 at 16:46 Pepelisu ***@***.***> wrote:
Dear Ruud,
The error seems to be that the update of the node failed somehow and some
files are not where they should be, otherwise I am not sure why the logs
are showing import errors.
Could you try to reinstall the ethoscope-node package as suggested by
ggilestro?
—
Reply to this email directly, view it on GitHub
<#178 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABSQSRX4WVEGOCRMNKFUEILXH7AKDANCNFSM6AAAAAAUV5GDLU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Small update: Instead of "yay" (which does not seem to be installed by default) I used "pacman -S ethoscope-node" to reinstall the package, after first performing a "pacman -R ethoscope-node" removal. After this some progress I think: I can now initiate a tracking event, and manually (i.e. via the browser interface) force a backup to the node ethoscope-data folder successfully. The automated backup feature in the browser interface still is inactive, and once activated, it will become inactive again when the browser is refreshed. I am going to test if it may just take a very long time before the automated backup of tracking data is performed, but not holding my breath, as in older versions of the system the auto-backup was very fast after the tracking events were ended. Do you have any more ideas of what may be going on? |
|
To debug what is happening now, after the reinstall. We would need at least the output of: (in the node) and maybe, for more info: |
|
Dear pepelisu, sudo systemctl status ethoscope_backup: sudo systemctl status ethoscope_video_backup: journalctl -u ethoscope_backup -rb: I looked in the /opt/ethoscope-node/scripts folder and did find the .service files, so perhaps there is something in the way these files are referenced that is wrong? |
|
Make sure you use underscores. it's |
|
Apologies, should not have copy/pasted; see attached for the requested
outputs of the three commands.
-Ruud
…On Fri, May 26, 2023 at 10:21 AM Giorgio Gilestro ***@***.***> wrote:
Make sure you use underscores. it's ethoscope_backup not ethoscope-backup
—
Reply to this email directly, view it on GitHub
<#178 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABSQSRWYMP4LK3L73Q5IWGTXIC35FANCNFSM6AAAAAAUV5GDLU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
seems the attachments didn't come through. |
|
Some things to try:
The error that your logs are showing is solved in the last version of ethoscope node (branch dev) |
|
Hi pepelisu, {"001c7cf3397741449c529aff01a294fb": {"status": "stopped", "ping": 2246, "ip": "192.168.0.100", "time": 1685062269.7118032, "error": null, "log_file": "/ethoscope_data/results/ethoscope.log", "dbg_img": "/ethoscope_data/results/dbg_img.png", "last_drawn_img": "/tmp/ethoscope_ba10yz9r/last_img.jpg", "db_name": "ETHOSCOPE_001_db", "monitor_info": {"last_positions": null, "last_time_stamp": 0, "fps": 0}, "experimental_info": {"run_id": "8b4184203893417b"}, "id": "001c7cf3397741449c529aff01a294fb", "name": "ETHOSCOPE_001", "version": {"id": "3312673383686b246be8885c5913c3a69ecc6049", "date": "2023-02-06 11:46:58"}, "backup_filename": "2023-05-26_00-49-14_001c7cf3397741449c529aff01a294fb.db", "CPU_temp": 41.3, "underpowered": false, "current_timestamp": 1685144781.726389, "backup_path": "/ethoscope_data/results/001c7cf3397741449c529aff01a294fb/ETHOSCOPE_001/2023-05-26_00-13-49/2023-05-26_00-13-49_001c7cf3397741449c529aff01a294fb.db", "backup_status": 100.0, "time_since_backup": 76557.93261098862, "backup_size": 147456, "previous_date_time": 1685060029.0185347, "previous_backup_filename": "2023-05-26_00-13-49_001c7cf3397741449c529aff01a294fb.db", "previous_user": "rschilder", "previous_location": "incubator"}} |
|
If the |
|
I was able to reproduce the issue on a new installation. The commit above should fix it. Please make sure you run this latest version of the node and the backup should work again. |
|
Dear Giorgio,
I am running the latest 06.07.23 version of the node now, but I am still
not seeing any tracking data backed up to the node.
I didn't try the manual backup procedure yet.
What can I log to help diagnose?
-Ruud
…On Wed, Jun 7, 2023 at 4:19 PM Giorgio Gilestro ***@***.***> wrote:
I was able to reproduce the issue on a new installation. The commit above
should fix it. Please make sure you run this latest version of the node and
the backup should work again.
—
Reply to this email directly, view it on GitHub
<#178 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABSQSRXEEQWGFQNTNWHGNNDXKDO4JANCNFSM6AAAAAAUV5GDLU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
The bug was in the backup service. Just make sure you restart the backup service after yesterday's upgrade. |
|
Thanks Giorgio,
"systemctl restart ethoscope_backup.service"
did the trick.
Thanks all for the support!
-Ruud
…On Thu, Jun 8, 2023 at 2:40 AM Giorgio Gilestro ***@***.***> wrote:
The bug was in the backup service. Just make sure you restart the backup
service after yesterday's upgrade.
—
Reply to this email directly, view it on GitHub
<#178 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABSQSRQQU4SEISJEX3CLOKLXKFXWZANCNFSM6AAAAAAUV5GDLU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Greetings all,
As you can see in the attached images, I have encountered problem related with issue number #105 #127 and #149 . The problem started after I updated the node. Description of the problem:
The version of the node:
As it was not allowing automatic backup, I tried forcing it as shown here and this was the result:
`cd /opt/ethoscope-node/node_src/scripts
sudo python backup_tool.py -e 1
Forcing backup for ethoscope 001
INFO:root:Initiating backup for device 001d54d8d1c94e43979d5805c9d995a7
INFO:root:Running backup for device 001d54d8d1c94e43979d5805c9d995a7
INFO:mysql.connector.authentication:package: mysql.connector.plugins
INFO:mysql.connector.authentication:plugin_name: mysql_native_password
INFO:mysql.connector.authentication:AUTHENTICATION_PLUGIN_CLASS: MySQLNativePasswordAuthPlugin
INFO:root:Initializing local database static tables at /ethoscope_data/results/None
INFO:root:Making parent directories
WARNING:root:[Errno 17] File exists: '/ethoscope_data/results'
INFO:root:Ensuring DAM file exists at /ethoscope_data/results/None.txt
INFO:root:Database mirroring initialised
ERROR:root:Cannot mirror the 'SENSORS' table
ERROR:root:1146 (42S02): Table 'ETHOSCOPE_001_db.SENSORS' doesn't exist
INFO:root:Backup status for 001d54d8d1c94e43979d5805c9d995a7 is 100.00%
INFO:root:Backup done for for device 001d54d8d1c94e43979d5805c9d995a7 `
Then when I checked the /ethoscope_data/results directory, it has saved the data as NONE.db and NONE.txt
Also, I tried checking the status of the ethoscope_backup.service in systemctl and this was the result:
`● ethoscope_backup.service - Ethoscope node server
Loaded: loaded (/usr/lib/systemd/system/ethoscope_backup.service; enabled; preset: disabled)
Active: active (running) since Wed 2023-02-08 18:10:57 CST; 5min ago
Main PID: 429 (python)
Tasks: 1 (limit: 18983)
Memory: 35.3M
CPU: 231ms
CGroup: /system.slice/ethoscope_backup.service
└─429 /usr/bin/python /opt/ethoscope-node/node_src/scripts/backup_tool.py
Feb 08 18:11:03 node python[429]: urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
Feb 08 18:11:03 node python[429]: During handling of the above exception, another exception occurred:
Feb 08 18:11:03 node python[429]: Traceback (most recent call last):
Feb 08 18:11:03 node python[429]: File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
Feb 08 18:11:03 node python[429]: self.run()
Feb 08 18:11:03 node python[429]: File "/opt/ethoscope-node/node_src/ethoscope_node/utils/backups_helpers.py", line 148, in run
Feb 08 18:11:03 node python[429]: active_devices = self.find_devices()
Feb 08 18:11:03 node python[429]: File "/opt/ethoscope-node/node_src/ethoscope_node/utils/backups_helpers.py", line 89, in find_devices
Feb 08 18:11:03 node python[429]: logging.error("The node ethoscope server %s is not running or cannot be reached. A list of available ethoscopes could not be found." % server)
Feb 08 18:11:03 node python[429]: NameError: name 'server' is not defined`
As I am a beginner, I wanted to ask if you guys can help me on how to resolve this problem. I have several experiments lined up, so urgent help is much appreciated.
Thanks
The text was updated successfully, but these errors were encountered: