Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modem vanishes randomly #185

Closed
justinmbrock opened this issue Jun 26, 2021 · 23 comments
Closed

Modem vanishes randomly #185

justinmbrock opened this issue Jun 26, 2021 · 23 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@justinmbrock
Copy link

I am running the Phosh build on a 3GB PinePhone 1.2a modded to 1.2b.
Several other maintainers are troubleshooting issues with the modem disappearing (ex: https://invent.kde.org/teams/plasma-mobile/issues/-/issues/3). When the phone sleeps, the modem may or may not work when the phone wakes up, and restarting eg25-manager or resetting the usb port usually fixed it for most, but that's just a bad hack.

I've had this problem intermittently for months, but the last few weeks the phone doesn't even need to deep sleep for the modem to randomly disappear, plugged in or not. Restarting eg25-manager does not "fix" anything, and to get anything through the modem again I have to reboot the phone.

Here is something else I noticed. I just rebooted. Here is the output of ls /dev/ttyUSB* and lsusb:

[alarm@pinephone][~]% ls /dev/ttyUSB*
/dev/ttyUSB0  /dev/ttyUSB1  /dev/ttyUSB2  /dev/ttyUSB3
[alarm@pinephone][~]% lsusb
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
**Bus 002 Device 002**: ID 2c7c:0125 Quectel Wireless Solutions Co., Ltd. EC25 LTE modem
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

And about 2 minutes later after the modem vanished:

[alarm@pinephone][~]% ls /dev/ttyUSB*
/dev/ttyUSB0
[alarm@pinephone][~]% lsusb
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
**Bus 002 Device 004**: ID 2c7c:0125 Quectel Wireless Solutions Co., Ltd. EC25 LTE modem
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

The modem is a different device number now, in addition to the serial ports missing. If you want anything else let me know what you want and how to get it. If this is better addressed upstream, please point me in the right direction, and I'll go there. Thanks!

@Danct12
Copy link
Member

Danct12 commented Jun 27, 2021

Got dmesg? It's possible that the OS inside the modem crashed, which firmware do you use?

@Danct12 Danct12 added the bug Something isn't working label Jun 28, 2021
@Danct12
Copy link
Member

Danct12 commented Jun 28, 2021

Tomas Oqvist reported the same problem on Telegram, the user claimed that most of the modem vanishing caused while the phone is charging.

There seems to be no sign of USB EMI when the modem is lost. Once the modem is lost, eg25-manager daemon tries to reboot the modem but failed to do so.

jun 28 16:28:55 danctnix eg25manager[3334]: Lost modem, resetting...
jun 28 16:28:55 danctnix eg25manager[3334]: Couldn't unbind modem: wrote -1/3 bytes
jun 28 16:28:55 danctnix eg25manager[3334]: Couldn't bind modem: wrote -1/3 bytes

A few seconds later, a mysterious USB device appears on the bus which ModemManager doesn't know how to use it.

jun 28 16:29:02 danctnix kernel: usb 2-1: new high-speed USB device number 3 using ehci-platform
jun 28 16:29:02 danctnix kernel: option 2-1:1.0: GSM modem (1-port) converter detected
jun 28 16:29:02 danctnix kernel: usb 2-1: GSM modem (1-port) converter now attached to ttyUSB0
jun 28 16:29:02 danctnix systemd-udevd[5082]: 2-1: /usr/lib/udev/rules.d/80-modem-eg25.rules:3 Failed to write ATTR{/sys/devices/platform/soc/1c1b000.usb/usb2/2-1/power/wakeup}, ignoring: No such file or directory
jun 28 16:29:06 danctnix systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
jun 28 16:29:06 danctnix audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
jun 28 16:29:06 danctnix kernel: audit: type=1131 audit(1624890546.501:83): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=NetworkManager-dispatcher comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
jun 28 16:29:10 danctnix ModemManager[3495]: <info>  [device /sys/devices/platform/soc/1c1b000.usb/usb2/2-1] creating modem with plugin 'quectel' and '1' ports
jun 28 16:29:10 danctnix ModemManager[3495]: <warn>  [plugin/quectel] could not grab port ttyUSB0: Cannot add port 'tty/ttyUSB0', unhandled port type
jun 28 16:29:10 danctnix ModemManager[3495]: <warn>  [base-manager] couldn't create modem for device '/sys/devices/platform/soc/1c1b000.usb/usb2/2-1': Failed to find primary AT port

I personally think that there is something wrong with the software running on the modem as I have never seen this strange behavior before and I have no way to reproduce the issue on my end.

Modem software version: EG25GGBR07A08M2G_01.002.01.002

@Danct12 Danct12 added the help wanted Extra attention is needed label Jun 28, 2021
@lazmol
Copy link

lazmol commented Jun 28, 2021

Hello,
have the same problem but I am on Biktorgj's 0.2.1 modem. Just wanted to try going back to the vendor modem firmware but it seems the problem occurs there too...
I see it more often when the phone is hot (e.g. when charging, like mentioned above).
Attached the output of journalctl -b | grep -i "eg25\|modem"
journalctlb_modemfail_grepeg25modem.log

I might update to Biktorgj's latest firmware version and check.
Pls tell me if I can help somehow debugging this problem.

@justinmbrock
Copy link
Author

Attached is a dmseg log from a reboot that ended with the modem vanishing after the screen timed out once. I'm running EG25GGBR07A08M2G_01.002.07 per the wiki instructions here.

dmesg.log

@Michasze
Copy link

Michasze commented Jul 3, 2021

Had the same problem. Downgrading eg25-manager to 0.3.0+2+g64145ac fixed the issue for me.

@returnofthelambda
Copy link

I had the same issue on a fresh install of the 6/28 build. Cell modem would disappear every couple of hours. Disabling the wifi/bluetooth via the physical kill-switch helped delay the issue, but did not fix it. I enabled the wifi/bluetooth toggle and then I tried downgrading eg25-manager as suggested by @Michasze. It also helped keep the modem connected longer, but the modem would still disappear after awhile. Running pacman -Syu last night pulled a new kernel (5.12.14-1) and I haven't had any issues since. I am still using eg25-manager version 0.3.0+2+g64145ac from the 6/13 build.

@kescherCode
Copy link

Jul 07 15:20:45 pinephone systemd[1]: Started Quectel EG25 modem.
Jul 07 15:20:45 pinephone eg25manager[3588]: Opening config file: /usr/share/eg25-manager/pine64,pinephone-1.2.toml
...
Jul 07 15:21:06 pinephone eg25manager[3588]: ModemManager object `/org/freedesktop/ModemManager1/Modem/0' added
...
Jul 07 17:14:33 pinephone eg25manager[3588]: ModemManager object `/org/freedesktop/ModemManager1/Modem/0' removed
Jul 07 17:14:33 pinephone eg25manager[3588]: Lost modem, resetting...
Jul 07 17:14:33 pinephone eg25manager[3588]: Couldn't unbind modem: wrote -1/3 bytes
Jul 07 17:14:33 pinephone eg25manager[3588]: Couldn't bind modem: wrote -1/3 bytes
Jul 07 17:14:39 pinephone eg25manager[3588]: ModemManager object `/org/freedesktop/ModemManager1/Modem/1' added
...
Jul 07 19:55:06 pinephone eg25manager[3588]: Lost modem, resetting...
Jul 07 19:55:06 pinephone eg25manager[3588]: Couldn't unbind modem: wrote -1/3 bytes
Jul 07 19:55:06 pinephone eg25manager[3588]: Couldn't bind modem: wrote -1/3 bytes
Jul 07 19:55:06 pinephone eg25manager[3588]: ModemManager object `/org/freedesktop/ModemManager1/Modem/1' removed
Jul 07 19:55:12 pinephone eg25manager[3588]: ModemManager object `/org/freedesktop/ModemManager1/Modem/2' added

Looks like ModemManager knows how to handle the reconnecting modem after all... But it will not continue resetting the modem on every failure. At some point, it will not add another modem.

Jul 08 05:24:44 pinephone eg25manager[3588]: Lost modem, resetting...
Jul 08 05:24:44 pinephone eg25manager[3588]: Couldn't unbind modem: wrote -1/3 bytes
Jul 08 05:24:44 pinephone eg25manager[3588]: Couldn't bind modem: wrote -1/3 bytes
Jul 08 05:24:44 pinephone eg25manager[3588]: ModemManager object `/org/freedesktop/ModemManager1/Modem/6' removed

No further output (into the journal) happens. And this is when we see the modem never come back from vanishing (and wwan, telephony being completely unavailable). So, the modem is disappearing more often than we think it is, but ModemManager and eg25-manager seem to be recovering a lot.

However, when we restart, we see the following output:

Jul 08 10:13:34 pinephone eg25manager[3588]: Request to quit...
Jul 08 10:13:34 pinephone systemd[1]: Stopping Quectel EG25 modem...
Jul 08 10:13:34 pinephone eg25manager[3588]: dropping systemd sleep delay inhibitor
Jul 08 10:13:34 pinephone eg25manager[3588]: Powering down the modem...
Jul 08 10:13:35 pinephone eg25manager[3588]: Executed power-on/off sequence
Jul 08 10:13:35 pinephone eg25manager[3588]: Executed power-off sequence
Jul 08 10:13:37 pinephone eg25manager[3588]: Modem down, quitting...
Jul 08 10:13:38 pinephone systemd[1]: eg25-manager.service: Deactivated successfully.
Jul 08 10:13:38 pinephone systemd[1]: Stopped Quectel EG25 modem.
...
Jul 08 10:13:38 pinephone systemd[1]: Started Quectel EG25 modem.
...
Jul 08 10:14:00 pinephone eg25manager[27082]: ModemManager object `/org/freedesktop/ModemManager1/Modem/7' added

At that point, the modem comes back and becomes usable again, as indicated by the cellular icon in the (in my case) Phosh statusbar.
This means that we are actually encountering two issues here:

  • The modem vanishing in the first place
  • It not being properly reset after having vanished a number of times

@1001111github
Copy link

I have had the same problem with the modem disappearing for the past month. I also had phosh login screen control issues. I religiously pacman -Syu daily. As of two days ago the problems stopped occurring. Other than the pacman updates the only thing I changed was my power options in Settings.

Previous settings were 10 minutes display, 15 minute suspend on battery. No suspend on power.
New settings were 10 minutes display, 15 minute suspend on battery. 1 hour suspend on power.

For two days the phone has been 100% reliable. No modem drops, no login screen issues
YMMV
Dave (aka lf)

@kescherCode
Copy link

I flashed the 01.002.07 modem firmware (which still shows up as 01.002.01.002, go figure). Apparently, that breaks GPS, but my modem has never gone away since. Same as Dave before me: YMMV.

@Danct12
Copy link
Member

Danct12 commented Jul 27, 2021

According to biktor, the only single /dev/ttyUSB device showing up after losing cellular is the modem crashed to EDL/Sahara mode.

Screenshot_20210727_161709

Sometimes, the modem successfully rebooted back to the OS and work again, then crashed sometime later.

There's also reports on Pine64 forum as well: https://forum.pine64.org/showthread.php?tid=13885

Mobian: https://gitlab.com/mobian1/issues/-/issues/251

@kescherCode
Copy link

I would like to add that the permanent vanishing seems to occur with modem firmwares 01.002.01.002 and 01.003.01.003, but I've flashed 01.002.07 (which still reports as 01.002.01.002 for some reason), and ever since, the modem still vanishes occasionally ("Lost modem, resetting..."), but always comes back within 10 seconds by itself, and never disappears permanently. Also, the vanishing occurs much less frequently.

(I don't suggest flashing that modem firmware though, it seems to be broken in a number of other ways)

@Jab2870
Copy link

Jab2870 commented Jul 27, 2021

@kescherCode could you elaborate on some of the other ways the 01.002.07 version of the modem is broken?

@kescherCode
Copy link

@Jab2870 well, AT+QGPS=1 returning +CME ERROR=504, for instance. And its version string is still returning 01.002.01.002.

Allegedly, there are more things that are broken in comparison to 01.002.01.002, but I don't really notice as the PinePhone isn't my daily driver.

@dukla2000
Copy link

dukla2000 commented Aug 10, 2021

IMHO the modem software revision is not a significant factor.
My phone (1.2b 3Gb mobo) dual boots between Mobian and Arch.
On Mobian I have no problems - well every week or 2 I may have a modem issue but it is relatively rare and I have no way of reliably recreating.
On Arch I typically run into a problem after an hour or 2.
Both installations are updated about daily:
Mobian is:
`$ apt search eg25
Sorting... Done
Full Text Search... Done
eg25-manager/unstable,now 0.3.0-1 arm64 [installed,automatic]
Manager daemon for the Quectel EG25 mobile broadband modem

eg25-manager-dbgsym/unstable 0.3.0-1 arm64
debug symbols for eg25-manager

csn@mobian:~$ uname -a
Linux mobian 5.13-sunxi64 #1 SMP PREEMPT Sat Aug 7 15:32:08 UTC 2021 aarch64 GNU/Linux`

Arch started with 20210713 version and is:
$ pacman -Ss eg25 danctnix/eg25-manager 0.3.0+6+gf2593b6-1 [installed] Daemon for managing the Quectel EG25 modem [csn@pparch ~]$ uname -a Linux pparch 5.13.8-1-danctnix #1 SMP Mon Aug 9 17:54:52 UTC 2021 aarch64 GNU/Linux

My vote would be something in the kernel compilation options (I think arch is more-megi and mobian is less-megi?) but I really am pretty damn ignorant so feel free to laugh at that suggestion. (And I have no axe to grind with megi, I use his p-boot for my dual boot!)

PS (2 days later)
I have been on 01.003.01.003 so I flashed 01.002.01.002 - first 36 hours didn't miss a beat. Last 6 hours have had 2 cases of lost modem again (on arch).

@Jab2870
Copy link

Jab2870 commented Aug 12, 2021

Is there anything (logs etc) I can provide to help with this. This is the only reason I can't use Arch on my pinephone as my Daily driver.

@1001111github
Copy link

As mentioned before I have been experiencing the modem dropping problem.

Last week it started happening every 30 minutes and the modem/phone would get wery hot.

Shortly after the modem was not always detected on boot. Now it is never detected on boot.
SDCard or EMMC does not matter, Arch, Manjaro and Mobian have been tried.
The hardware switch has been flipped multiple times.

[cone@pine ~]$ dmesg|grep odem
[ 0.279650] cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters
[ 13.714090] audit: type=1130 audit(1628878067.924:39): pid=1 uid=0 auid=4294967295 ses=4294967295 subj==unconfined msg='unit=ModemManager comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

[cone@pine ~]$ journalctl -b -u eg25-manager

-- Journal begins at Mon 2021-08-02 11:59:28 MDT, ends at Fri 2021-08-13 12:13:30 MDT. --
Aug 13 12:07:44 pine systemd[1]: Started Quectel EG25 modem.
Aug 13 12:07:44 pine eg25manager[3441]: Opening config file: /usr/share/eg25-manager/pine64,pin>
Aug 13 12:07:44 pine eg25manager[3441]: Starting modem...
Aug 13 12:07:46 pine eg25manager[3441]: Executed power-on/off sequence
Aug 13 12:07:46 pine eg25manager[3441]: taking systemd sleep inhibitor
Aug 13 12:07:46 pine eg25manager[3441]: oFono vanished from D-Bus
Aug 13 12:07:46 pine eg25manager[3441]: inhibitor sleep fd is 20
Aug 13 12:07:47 pine eg25manager[3441]: ModemManager appeared on D-Bus
Aug 13 12:07:56 pine eg25manager[3441]: Response: [RDY]
Aug 13 12:07:56 pine eg25manager[3441]: taking systemd sleep inhibitor (blocking)
Aug 13 12:07:56 pine eg25manager[3441]: inhibitor block fd is 21
Aug 13 12:07:56 pine eg25manager[3441]: Response: [+CFUN: 1]
Aug 13 12:07:57 pine eg25manager[3441]: Response: [+CPIN: NOT INSERTED]
Aug 13 12:09:56 pine eg25manager[3441]: Modem is up for 120 seconds and fully ready
Aug 13 12:09:56 pine eg25manager[3441]: dropping systemd sleep block inhibitor

I have not opened the phone to check the traces, yet... :(
Any ideas or suggestions?

@Roboron3042
Copy link

This happens to me as well recently @1001111github .I've found it happens more often if the phone is hot, so even if you restart it fails. You have to shut down the phone first, wait for the phone to cool down enough, then power on the phone again.

@kapinta
Copy link

kapinta commented Aug 15, 2021

Is there anything (logs etc) I can provide to help with this. This is the only reason I can't use Arch on my pinephone as my Daily driver.

I am in the same situtation, but I have been using pinephone with Arch as daily device for several months.
If I can help in any way or attach more logs please tell me how to do it or what to upload.
I my case, as told before, the modem does not recover if the mobile signal has been down for example when entering to a garage.
I forgot to say that I am using the first pinephone device, the one with 2GB RAM.
Thank you!

@arkeo-ca
Copy link

arkeo-ca commented Aug 25, 2021

Might be anecdotal at best, but I upgraded my Modem FW to the open-source Biktorgj 0.3.0 and it seems to have improved the situation tremendously.

I had the modem vanish twice on the first day (last Sunday) and could not get it to work by simply restarting the eg25-manager service (I had to reboot), but it has not vanished on me a single time since. Or if it has, it recovered before I had a chance to notice. Keeping my fingers crossed that its a long term solution, as I am using this phone as my daily driver and really need the reliability.

Edit (27 Aug 21): The modem still vanishes 1-2 a day with heavy use, but it is nowhere near as bad as it used to be. Baby steps, I guess...

@Danct12
Copy link
Member

Danct12 commented Feb 14, 2022

It seems like the discussion has died down, I haven't heard of anyone's modem resetting for months now.

Closing.

@Danct12 Danct12 closed this as completed Feb 14, 2022
@rothn
Copy link

rothn commented Apr 2, 2023

Can we reopen? This is definitely not fixed and has happened at least twice today!

@kescherCode
Copy link

kescherCode commented Apr 2, 2023

@rothn Are you using the official firmware? If so, try switching to Biktorgj's firmware. It's much better and doesn't crash all the time. Yep, the modem's official firmware seems to be at fault with this.

@rothn
Copy link

rothn commented May 26, 2023

I have been using Biktorgj's firmware without any issues since your comment until today, when the modem disappeared and did not reappear. Rebooting did not fix it. I ultimately had to take the battery out, wait 10 seconds, and put it back in before the modem started working again.

Still, 2 months uptime sure beats it dropping (and not coming back) multiple times per day.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests