Log human-readable SCSI error code data instead of hexadecimal values #20
Comments
uweseimet
changed the title
|
@supercompman @joeventura1 @jcres69 @edgeux Some time ago in PiSCSI/piscsi#656 you stumbled upon cases where PiSCSI was reporting "Not able to receive 512 bytes of data". Because of numerous error handling and logging improvements as part of the new SCSI2Pi project (see https://www.scsi2pi.net/en/scsi2pi.html for details), it might be possible now to find out more about what's actually happening. If you would like to help with further analyzing this problem, please first verify whether it still exists in PiSCSI 23.11.01 and/or in SCSI2Pi 1.1.1. The latter you can download as a binary package on https://www.scsi2pi.net/en/downloads.html, i.e. compiling the sources is not required. I would appreciate your feedback. |
|
I reimaged my pi4b and loaded the latest piscsi 23.11.1 and can confirm that the issue still exists on my mac plus. Let me know how I can help…
Jan 17 19:13:47 piscsi PISCSI[503]: [2024-01-17 19:13:47.651] [info] Validating: operation=ATTACH, command params='locale=en', 'token=???', device=6:0, >Jan 17 19:13:47 piscsi PISCSI[503]: [2024-01-17 19:13:47.651] [info] Executing: operation=ATTACH, command params='locale=en', 'token=???', device=6:0, t>Jan 17 19:13:47 piscsi PISCSI[503]: [2024-01-17 19:13:47.652] [info] Attached SCHD 6:0
Jan 17 19:13:55 piscsi systemd[1]: Starting Online ext4 Metadata Check for All Filesystems...
Jan 17 19:13:55 piscsi systemd[1]: e2scrub_all.service: Succeeded.
Jan 17 19:13:55 piscsi systemd[1]: Finished Online ext4 Metadata Check for All Filesystems.
Jan 17 19:14:17 piscsi systemd[1]: systemd-hostnamed.service: Succeeded.
Jan 17 19:14:37 piscsi dhcpcd[683]: eth0: Router Advertisement from fe80::baf8:53ff:fef7:e161
Jan 17 19:16:37 piscsi kernel: sched: RT throttling activated
Jan 17 19:16:39 piscsi PISCSI[503]: [2024-01-17 19:16:39.505] [error] (ID 6) - Not able to receive 512 byte(s) of data, only received 179
|
|
@jcres69 Thank you for your feedback and for offering your help. Because the issue still exists with PiSCSI 23.11.01 I guess it also exists with SCSI2PI, but the next step would be to confirm this. Can you please install SCSI2Pi, either by using the binary packages offered on https://www.scsi2pi.net/en/downloads.html, or by cloning the GitHub sources on https://github.com/uweseimet/scsi2pi? Using the binaries is faster, but for the purpose of testing it is better to compile the sources of the develop branch, and with a Pi 4 the compilation time is quite acceptable. You do not have to compile everything, "make s2p" is sufficient. If you find that s2p reports the same issues as piscsi please drop me a note in this ticket. I will then create a new issue, just for this purpose. |
|
I'm sort of a noob, so forgive me if I make some dumb mistakes.
I downloaded and installed "scsi2pi_1.2_devel_fb4e56b_debug_arm64-1.deb"
But, when I try to run s2p or s2pctl, I get this error message:
***@***.***:~ $ s2p
s2p: error while loading shared libraries: libprotobuf.so.32: cannot open shared object file: No such file or directory
***@***.***:~ $ s2pctl
s2pctl: error while loading shared libraries: libprotobuf.so.32: cannot open shared object file: No such file or directory
|
|
@jcres69 Which Pi are you using, and which OS are you running? What does 'lsb_release -a' say, and what does 'ldd s2p' say? |
|
Pi4b, Raspberry Pi OS lite (64 bit)
***@***.***:~ $ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
|
|
@jcres69 Please also provide the output of 'ldd s2p' and 'ldd piscsi' (from the PiSCSI project) on the same machine. I have tested the binaries with the regular Pi OS, but not with "lite". Looks as if there are special requirements for the libraries that have to be installed. |
|
@jcres69 By the way, if it is possible, please do not add comments to GitHub by email, but in the GitHub web UI. Otherwise the comments in the tickets are quite hard to read and contain a lot of noise unless you edit them like I just did. |
|
@jcres69 I think I know now what the issue might be: You downloaded the 64 bit binary, which I (at least currentl) only provide for bookworm (also see the description on the download page). Does the 32 bit binary (which is for both bullseye and bookworm) work for you? |
|
@uweseimet I will try the 32bit binary. and here is the LDD output for both scsi2pi and piscsi. yes, it looks like there are some things missing on the OS lite, or I just installed the wrong binary... $ ldd /usr/local/bin/piscsi $ ldd /opt/scsi2pi/bin/s2p |
|
@jcres69 Thank you for the output. I will analyze it and then decide whether I have to provide yet one more binary package. This will also depend on whether the 32 bit binaries work for you. |
|
@jcres69 I almost forgot: When installing s2p the original piscsi is replaced by a link to s2p and backed up. This is required in order to still have a binary called 'piscsi', because the PiSCSI system service needs it. |
|
@uweseimet I installed the 32bit binary, but i get this error. I'm thinking i should just reimage and try again with the full Pi image... pi@piscsi:~ $ sudo dpkg -i scsi2pi_1.2_devel_fb4e56b_armhf-1.deb |
|
@jcres69 You can ignore this error message. As far as I can tell you get it because the previous scsi2pi installation was not de-installed (see the scsi2pi download webpage for the de-installation command), but this is not a problem. I will try to address this particular problem anyway. The binary should be properly installed, because the error message just referred to the manpage. Can you please run 'ldd /opt/scsi2pi/bin/s2p'? Do you know how to compile PiSCSI from the sources? If you do, the procedure for SCSI2Pi is more or less the same. While trying to work on the "Could not receive" issue compiling from the sources would be required anyway, because I may have to change the source code and then I would like to ask you to recompile and test the change. This incremental approach cannot really be done with binaries. |
|
ldd /opt/scsi2pi/bin/s2p I can clone the github repo, and compile it if that is what you mean, but i'd need instructions to make sure i'm doing it right. This is what i tried to do first with s2p, but it didnt run after completing, so I installed the .deb package. |
|
@jcres69 Can you please run 'ls -al /opt/scsi2pi/bin/s2p'? I would like to ask you not to replace your lite installation by a different one, if possible. During the weekend I will try to install Pi os lite on my Pi, so that I can better analyze which kind of binary is needed. |
|
@uweseimet |
|
@jcres69 When you try to run s2p in this folder, which error message do you get? |
|
@uweseimet |
|
@jcres69 That's strange, ls does list this file, doesn't it? I mean, the file is there, in the folder? |
|
@uweseimet yes it's there, just wont run! |
|
@jcres69 OK, I will install Pi OS lite on my Pi and test tomorrow. So at least for now compiling appears to be the only option. Which error message do you get? |
|
@uweseimet I don't remember seeing any errors during compile, but s2p didnt seem to be running so I thought I made a mistake, and then just installed the deb package (which turned out to be the wrong one). |
|
@jcres69 After compiling s2p you can launch it manually. It's located in cpp/bin/s2p. This will definitely work. Or you just replace your piscsi binary in /usr/local/bin with it. But you cannot run the piscsi binary and the s2p binary at the same time, because they both try to use the same resources. |
|
@uweseimet yes that seems to work. does this look good? pi@piscsi:~/scsi2pi/cpp/bin $ sudo ./s2p |
|
@jcres69 Yes, that looks good, indeed! Remember (also see the information on the SCSI2Pi website), s2p is essentially a replacement for the piscsi binary. This means after replacing piscsi with s2p your PiSCSI installation should work as before, except for the numerous new features/fixes listed on the SCSI2Pi website. This will give you version 1.2-devel, and that's the one where based on your test results (logfiles etc.) I can try to apply fixes to. But first we need to know whether 1.2-devel still reports the error message. |
|
@uweseimet ahhh, right I didnt checkout the develop branch. ok, I will do that and report back after testing... |
|
@jcres69 Cool, thank you! As soon as you have a working setup things get rather easy: I will (hopefully, i.e. I hope that I will find out what's wrong) change the sources, you will update your checked out sources, recomple ('make') , and test. This is an incremental process, which may take some time. |
This was due to a typo, which is just fixed. |
|
@jcres69 When you resume testing please update (git pull) your develop branch and recompile. Note that the offending message has changed to something like "Received x bytes(s) in DATA OUT phase, expected to receive y" and is now on warning level. |
|
how do I enable trace logging? |
|
With "-L trace" as additional option when lauching s2p, e.g. I recommand first checking whether the message appears at all with the standard loglevel, because trace can produce a lot of noisy output. |
|
@uweseimet so far I've been unable to reproduce the error, however when I switch to piscsi it is very easy to reproduce. I enabled trace logging in piscsi and here are a few logs showing the error in case this helps. Also, I noticed that "piscsi kernel: sched: RT throttling activated" always occurs in syslog prior to the error. Again, I have not seen any of this using s2p. [2024-01-21 11:26:07.810] [trace] (ID 5) - Receiving data, transfer length: 512 byte(s) [2024-01-21 11:26:13.872] [trace] (ID 5) - Phase: dataout [2024-01-21 15:44:05.095] [trace] (ID 6) - Bus Free phase |
|
@jcres69 Thank you for testing. If this problem is really gone with SCSI2Pi this would be because of the extensive code cleanup and many improvements I applied when setting up the SCSI2Pi project. But whether it is really gone only time will tell, because this it appears to have been quite spurious. On my Atari I only saw it twice in more than a year, this is why I asked for support from PiSCSI users who can more reliably reproduce this issue. I cannot yet comment on the "sched: RT throttling activated" message. I have to admit I have never looked into the system log, but I will do this in the future. It might be, though, that this message is not related to piscsi at all. Do you also see this message in this message in the logs when running s2p. I'm asking this because my changes included scheduling changes, indeed. By the way, soon I will also provide binary packages for bullseye 64 bit, so that all in all all potential platforms (32/64 bit bullseye/bookworm) are covered. |
|
@jcres69 I forgot to ask: You did notice my note on the message text being different, and that the message is logged on warning level now (with the SCSI2Pi develop branch), didn't you. I'm just asking in order to ensure that we don't miss the obvious, i.e. the message still being there with a different appearence. |
@uweseimet yes, I did note the different message text and still have not seen it yet. I'll continue to test and let you know if I see the error again. Thanks for doing this, I am able to use my rascsi again! |
|
@jcres69 Thank you for the good news! I hope you will never stumble upon this message again ;-). |
|
@uweseimet unfortunately I am seeing this message again, but only after using s2p v 2.x. the 1.1.2 and 1.2 versions don't have this issue: pi@raspberrypi:/opt/scsi2pi/bin $ sudo ./s2p -v |
|
@jcres69 I guess I know why. I had to fix a timing issue, and I almost suspected that this might be related to your issue. But I'm afraid that I had not choice but to fix this issue. On the other hand this also means that it appears to be very clear now which piece of code directly correlates to your initial problem. |
|
@uweseimet hmm what am i doing wrong here: pi@raspberrypi:/opt/scsi2pi/bin $ sudo ./s2p -id 6 daynaport I created the piscsi_bridge, what else do i need to do? pi@raspberrypi:/opt/scsi2pi/bin $ ip add |
|
@jcres69 You stumbled upon an issue with the RC4 release candiate, which will be resolved in the next RC later today. |
|
@uweseimet I tested the 2.0 release, and no I don't get the SCSI error messages, however I did notice that in this version (and the 2.1 version also) when I disconnect the devices, the piscsi_bridge interface goes down and I lose connection. In the 1.x version, the bridge stayed up as I believe it should. |
|
@jcres69 OK, so it's the timing change which is also mentioned in the release notes for 2.0.1. (The timing is the same again as with piscsi.) I will create a new ticket for the investigation of this issue. |
|
@uweseimet Maybe I am doing this wrong, but the Pi is hard wired on eth0 (not wifi) and I manually created the bridge interface based on the piscsi instructions. the bridge interface connects to my internal lan, and is how I remotely connect to the Pi. when the bridge interface goes down, I lose my ssh connection. |
|
@jcres69 There is no need to do anything manually. s2p will set up the bridge automatically, and as far as I remember piscsi does the same. Please also see the tickets I mentioned. |
|
#71 is the follow-up ticket for the data transfer issue. |
In order to improve the readability of the logfiles the hexadecimal error code values in the s2p logs shall be extended by a human-readable string representation, based on the respective names in the SCSI specification.
The text was updated successfully, but these errors were encountered: