12v alert doesn't shut down OVMS

[biddster]

I have a 2017 leaf 30 black edition with ovms running the latest firmware.

I've calibrated the 12v battery using my DVM and I've setup OVMS to shutdown after 2 minutes when the battery level goes 0.4v below threshold.

However, I just keep getting 12v alerts in the android app and OVMS never shuts down (at least I don't think it does). I'd assumed that I'd get a single alert and wouldn't get another until 12v is restored when I power the car up again.

Apologies if I've misunderstood the behaviour. Happy to help debug etc if this is a legit issue.

[dexterbg]

As mentioned on the UI, the module wakes up from shutdown (actually deep sleep) once per minute. Currently the 12V deep sleep mode doesn't apply your configured threshold settings to the 12V check on wakeup, but rather uses fixed standard values. If the voltage seems to be restored it will proceed to a normal boot, possibly leading to a new alert once the system is up and can apply your actual settings.

You can see the voltage level the boot module reads during the wakeup using a USB terminal (there is no file logging at this stage). Look for a log line "12V level: ~…V" and the following line. Please collect & post readings from a few wakeups.

To be able to apply the user config on 12V wakeup, the calibration & threshold value needs to be added to the RTC RAM section, and the section must not be cleared on wakeup if still intact. I saw occasional memory corruptions on my deep sleep tests, so that may not catch all wakeups.

Regards,
Michael

[jollyjinx]

I'm wondering about the deepsleep myself. So I setup a lab test here with a lab power supply with modbus support.
I calibrated the OVMS to 12.6Volt. I changed my module to not startup the modem (that is a different problem), so data I get are from mqtt via wifi. Furhtermore I changed the shutdowntime to 2 minutes.

I started up the OVMS module with 13V , once fully running I let the voltage drop to 11 V. This is the power usage curve on just plain 11V:

The module is more in HIGH mode than actually sleeping - that's supposed ? 660-670 mWatts (61 mA) n power and 260 mWatt (24mA) in deep sleep mode.

And the system is really booting every time as you can see on the monotonic counter curve:

With cellular enabled it will get worse as the modem will draw quite some power when booting up.

[dexterbg]

That's expected behaviour, as 11V is, depending on the measurement variance, high enough to assume power has been restored.

As written above, you can see the voltage level as interpreted by the boot module logged during the boot process.

[jollyjinx]

11V is far below health from what I read online, but I'm no battery expert. Out of interest I set the drop voltage to 9 volts - the same outcome:

Left top: lab power supply switching from 13.00V to 9.00V
Left bottom: OVMS own reading via mqtt ( I calibrated to 12.6 V as that would be my normal shutdown voltage)
Right Top: power usage of OVMS
Right Bottom: monotonic counter of OVMS via mqtt

Furthermore it sends me every 5 minutes critical battery warning reading 8.8, 8.4 and 8.5V:

To me it looks like it's not really saving much power with the 1 minute sleeps. That's supposed that way ?

Is there anything I can test to help ?

[dexterbg]

You still didn't post your boot log 12V messages, so I cannot know what uncalibrated voltage level your module reads (see above).

[jollyjinx]

I did only once see a log with the 12 level but I no longer see them during boot on the console - why is that ? Is there a way to force it to log that ? Is my Terminal broken ( I'm using screen on a mac connected via usb 112500 baud )

Once I saw:
I (133) boot: 12V level: ~11.5V
I (136) boot: 12V level sufficient, proceeding with boot

which was at 11Volts real Voltage. So 9Volts should keep it down right ?

[dexterbg]

Yes, 9V should be classified as unbootable.

The 12V level is only checked & logged on a boot from a wakeup. You don't need to wait for the module to detect a low level, you can also use test sleep 3 to enter deep sleep for 3 seconds. That won't turn off components but should give you the wakeup boot process.

[jollyjinx]

I with the celluar modem disabled I got it somehow in a state where the module would do as expected - wake up briefly and then stay in deep sleep.

Then the module was showing the

I (133) boot: 12V level: ~9.1V

messages every time. I disconnected usb and it stayed that way. for 4 deep sleep cycles. Just to get back to normal mode and after 2 minutes go to deep sleep again.

Result so far: The module does enter deepsleep with 9V external battery ( module shows 9.1V) but wakes up as well with 9V.

Diagram: on the left 9v input voltage
on the right: about 10 Minutes - first in deep sleep with every minute a short blib to see if power is back , then full boot + 2 minutes before sleeping again - with a wakeup afterwards.

[jollyjinx]

Hmmm another finding - with the serial console attached it stays in deep sleep for an hour , detaching it , it will turn on the next time. (Doing 8V input voltage this time).

[dexterbg]

And the boot 12V messages when it returned to full boot were…?

[jollyjinx]

Well, I was not connecting to it at the time, so I don't know. I might have a look if anything got logged to the sd card when I'm back at my desktop.
The battery boot messages I only saw when it was coming from deep sleep on the serial console. When it is booting normally it skips that message - maybe my serial is too slow ? Can I set it to something higher than 112500 ?

[dexterbg]

See above on the log messages, serial speed isn't relevant. SD log won't help, as the early boot phase isn't logged to SD.

[jollyjinx]

I've done a grep through my scrollbackbuffer and I've seen that message only once where it had a sufficient battery level. But 50 times with insufficient battery level.

As another test to see if it would stay in deep sleep I did it leave running without a console attached last night: It continously booted with just 8V connected to obd and no serial console attached.
From the logs it boots and sees the 8V once it's up and running and sleeps 2 minutes after that ( I have it set to sleep within 2minutes of low battery in settings).

It looks like this the whole time:

tin> cat ovms.log.20221023*|egrep ' (boot|powermgmt):'
...
2022-10-23 05:43:45.667 CEST W (63257) powermgmt: 12V battery alert detected, will shutdown in 2 minutes
2022-10-23 05:45:45.667 CEST E (183257) powermgmt: Ongoing 12V battery alert time limit exceeded! Shutting down OVMS..
2022-10-23 05:45:45.667 CEST I (183257) boot: Shutting down for DeepSleep...
2022-10-23 05:45:45.797 CEST W (183387) boot: Shutdown: sdcard pending, 1 total
2022-10-23 05:45:45.797 CEST W (183387) boot: Shutdown: webserver pending, 2 total
2022-10-23 05:47:56.367 CEST W (63247) powermgmt: 12V battery alert detected, will shutdown in 2 minutes
2022-10-23 05:49:56.367 CEST E (183247) powermgmt: Ongoing 12V battery alert time limit exceeded! Shutting down OVMS..
2022-10-23 05:49:56.367 CEST I (183247) boot: Shutting down for DeepSleep...
2022-10-23 05:49:56.417 CEST W (183297) boot: Shutdown: sdcard pending, 1 total
2022-10-23 05:49:56.417 CEST W (183297) boot: Shutdown: webserver pending, 2 total
2022-10-23 05:52:06.997 CEST W (63247) powermgmt: 12V battery alert detected, will shutdown in 2 minutes
2022-10-23 05:54:06.997 CEST E (183247) powermgmt: Ongoing 12V battery alert time limit exceeded! Shutting down OVMS..
2022-10-23 05:54:06.997 CEST I (183247) boot: Shutting down for DeepSleep...
2022-10-23 05:54:07.117 CEST W (183367) boot: Shutdown: sdcard pending, 1 total
2022-10-23 05:54:07.117 CEST W (183367) boot: Shutdown: webserver pending, 2 total
2022-10-23 05:56:17.727 CEST W (63257) powermgmt: 12V battery alert detected, will shutdown in 2 minutes
...

I'm sorry if I'm not helpful, or if I did not understand you correctly. What else should I try ?

[jollyjinx]

Btw. I don't think I mentioned it, but the boot process always tells me about RTC_MODULE error about a lock being released before acquired.

I (51) command: Expanding DUKTAPE javascript engine
I (56) boot: Initialising BOOT (1100)
I (60) boot: Wakeup from deep sleep detected, wakeup cause 4
E (67) RTC_MODULE: /home/balzer/esp/esp-idf/components/driver/rtc_module.c:1528 (adc1_lock_release):adc1 lock release called before acquire
E (80) RTC_MODULE: /home/balzer/esp/esp-idf/components/driver/rtc_module.c:1528 (adc1_lock_release):adc1 lock release called before acquire
E (93) RTC_MODULE: /home/balzer/esp/esp-idf/components/driver/rtc_module.c:1528 (adc1_lock_release):adc1 lock release called before acquire
E (106) RTC_MODULE: /home/balzer/esp/esp-idf/components/driver/rtc_module.c:1528 (adc1_lock_release):adc1 lock release called before acquire
E (119) RTC_MODULE: /home/balzer/esp/esp-idf/components/driver/rtc_module.c:1528 (adc1_lock_release):adc1 lock release called before acquire
I (133) boot: 12V level: ~8.1V
E (136) boot: 12V level insufficient, re-entering deep sleep
ets Jul 29 2019 12:21:46

rst:0x5 (DEEPSLEEP_RESET),boot:0x1b (SPI_FAST_FLASH_BOOT)

[dexterbg]

The RTC_MODULE errors are expected and can be ignored as stated in the source, but the relevant log lines come directly before and after them:

I (60) boot: Wakeup from deep sleep detected, wakeup cause 4
I (133) boot: 12V level: ~8.1V
E (136) boot: 12V level insufficient, re-entering deep sleep

This tells you the boot module has correctly detected the wakeup cause and a too low 12V level, putting the module back to sleep. You should see this cycle once per minute after entering deep sleep.

You wrote your module leaves the cycle after some minutes. I need to know what you see exactly there when the module leaves the cycle and boots normally despite not having a high enough voltage.

My development module has now been doing the deep sleep cycle correctly for about an hour, so something's different with your module.

[jollyjinx]

I just looked at the source vehicle/OVMS.V3/main/ovms_boot.cpp cause I wanted to see if I could understand the error about the lock released before locked and saw the remark about the lock, so all good.

But looking at the code, I wonder: the battery code is only checked when the boot reason is DEEPSLEEP_RESET
so maybe it's not DEEPSLEEP_RESET but something else ?

Why not always check the battery, except for debugging purposes (so when a console is attached) ?
OVMS could also skip the battery check when its below a threshold as I see 1Volt readings when no ODB is attached ?

So my simple idea would be to

if batterylevel.isInRange(2,11)
{
   deepsleep()
}
else 
{ 
   continue booting
}

Furthermore I saw the level calculation using ints and floats without converting - that's doing alright with modern c compilers ?

On a side note there, can't OVMS store the calibration data somewhere that it's available on boot ?

[dexterbg]

Suspecting a different boot reason was what lead me to asking you to check the log.

Inserting the test only on deep sleep wakeup is done intentionally to reduce potential harm from a general issue with the ADC reading. That may need to be changed if your issue turns out to be common.

On the float: strictly, a cast to float for the adc_level would be better here, so the compiler can optimize the division into one, if it doesn't do so already. It won't change the outcome in any relevant way though.

[jollyjinx]

is there not a way to keep the boot messages somewhere in ram until it's clear if the log is written to file and then either dicarded (no logfile) or send directly to the log file.
That way it might be easier to have a look in the first boot stages.

Thinking of it , I would change the boot check in the following way as it would always protect the battery no matter what:

if batteryLevelisInUnhealthyRange && noSerialConsoleIsConnected 
{
    doDeepSleep()
}

On a side question - I read that the ESP deep sleep:
"The chip consumes around 0.15 mA (if the ULP coprocessor is on) to 10µA."

What else does cost so much power with OVMS when in deep sleep ?

[dexterbg]

Keeping the boot log wasn't necessary before, but you're of course welcome to implement that.

On changing the boot process: as written, that's nothing we should do without good reason, as it may lead to modules not being able to boot. Please try to get those log messages.

On the power consumption: the OVMS consists of more than the ESP32 core, most secondary or passive components cannot be switched off or put to sleep.

[jollyjinx]

Just FYI (EDIT now 4boxes)

I've looked at the now 4 ovms boxes ( 2021-03-25, 2022-03-05, 2022-05-23) I have here and see that they are having the same error curve on detecting the correct voltage ( measured / real voltage ) with the default setting of 195.7.

as I see that the voltage error is a almost linear function between 10-15 volts I added a linear error correction and with that the errors are now linear in that range:

so that one can get much better voltage estimation with just one correct measurement.

This would be needed for a good deep sleep that works at a specific voltage.
I will look into it once I find a way to get ovms development environment in docker running on my arm64 machines.

[IamMattM]

Deep sleep not held correctly without USB_5V feeding the system !!

I have just acquired an OVMS HW v3.3 with MODEM 7600G

I am running "edge" firmware and can concur with the findings from @jollyjinx that *the deep sleep mode only works when the (USB_5V) power rail is present.

I am testing with a bench power supply with my new kit before taking the risk to attach it to my EV vehicle AUX battery...

Has this issue not been resolved / understood since first posted here in 2022 ?

I have had a look at ther schematics for the main board and the modem board to try to see if I could figure out what could be the issue...CSL_DTR/RTS outputs from CP2102 (USB_5V powered) idling high in the precence of USB_5V and somewhat controlling ESP32_EN might relate...?

Obviously without <USB_5V> present one cannot get the boot messages out of the usb connector as the USB_2_Serial chipset would be unpowered.

I guess that those messages would come out directly out of the ESP32 at 3v3 if there was a clear Testpoint to attach to :
ESP32.TX0=ESP32.PAD[35]
or
CP2102.RXD
or
R25 , 12K ? (back feeding problems ? )

[IamMattM]

When it works properly with the <USB_5V> present:

I (201) boot: 12V level: ~11.4V
E (205) boot: 12V level insufficient, re-entering deep sleep
ets Jul 29 2019 12:21:46

rst:0x5 (DEEPSLEEP_RESET),boot:0x1f (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:4796
load:0x40078000,len:0
load:0x40078000,len:14896
entry 0x40078d74
I (1131) psram: This chip is ESP32-D0WD
I (1131) spiram: Found 64MBit SPI RAM device
I (1131) spiram: SPI RAM mode: flash 40m sram 40m
I (1134) spiram: PSRAM initialized, cache is in low/high (2-core) mode.
I (1142) cpu_start: Pro cpu up.
I (1146) cpu_start: Application information:
I (1150) cpu_start: Project name:     ovms3
I (1155) cpu_start: App version:      3.3.003-651-ge4bb89cc
I (1162) cpu_start: Compile time:     Feb 26 2024 00:00:21
I (1168) cpu_start: ELF file SHA256:  a3ecf7069755aafb...
I (1174) cpu_start: ESP-IDF:          v3.3.4-849-g6e214dc33
I (1180) cpu_start: Starting app cpu, entry point is 0x40081744
I (0) cpu_start: App cpu up.
I (2058) spiram: SPI SRAM memory test OK
I (2058) heap_init: Initializing. RAM available for dynamic allocation:
I (2058) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM
I (2065) heap_init: At 3FFBFEE8 len 00020118 (128 KiB): DRAM
I (2071) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM
I (2077) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM
I (2084) heap_init: At 4009CCC4 len 0000333C (12 KiB): IRAM
I (2090) cpu_start: Pro cpu start user code
I (2095) spiram: Adding pool of 4096K of external SPI memory to heap allocator
I (102) ovms_main: Set default logging level for * to INFO
I (102) ovms_main: WATCHDOG already initialized...
I (104) ovms-duktape: Initialising DUKTAPE Registry (1000)
I (110) command: Initialising COMMAND (1010)
I (116) command: Expanding DUKTAPE javascript engine
I (121) boot: Initialising BOOT (1100)
I (125) boot: Wakeup from deep sleep detected, wakeup cause 4
E (134) RTC_MODULE: /home/openvehicles/build/esp-idf/components/driver/rtc_module.c:1528 (adc1_lock_release):adc1 lock release called before acquire
E (147) RTC_MODULE: /home/openvehicles/build/esp-idf/components/driver/rtc_module.c:1528 (adc1_lock_release):adc1 lock release called before acquire
E (161) RTC_MODULE: /home/openvehicles/build/esp-idf/components/driver/rtc_module.c:1528 (adc1_lock_release):adc1 lock release called before acquire
E (175) RTC_MODULE: /home/openvehicles/build/esp-idf/components/driver/rtc_module.c:1528 (adc1_lock_release):adc1 lock release called before acquire
E (189) RTC_MODULE: /home/openvehicles/build/esp-idf/components/driver/rtc_module.c:1528 (adc1_lock_release):adc1 lock release called before acquire
I (201) boot: 12V level: ~11.4V
E (205) boot: 12V level insufficient, re-entering deep sleep

and over and over drawing 0.22W (11.4V@20mA) for 60secs and then briefly waking up and drawing 0.34W(11.4V@30mA)

[IamMattM]

My CONFIG/VEhicle/12Vmonitor is set to :

12V reference: 12.8V
12V threshold (0.5V) (i.e 12.3V)

12V shutdown: 12.0V
12V wakeup: 12.4V

[IamMattM]

I'll attempt to to place a <USB_2_SERIAL>@3v3 on resistor R25 (ESP32.TX0) , located in the top-end corner of the main PCB:

https://github.com/openvehicles/Open-Vehicle-Monitoring-System-3/blob/master/vehicle/hardware/v3.3/Mainboard_PCB_top_v3.3_20211213.pdf

This should clarify what the boot messages are when BAT=11.4V and the USB_5V is not present in the system : as I said earlier, for me, withoput USB_5V present, , the deep-sleep is NOT maintained, and the unit reboots fully , drawing 1.7W peak at some point (modem enabled) and will repeat this same full power-up over and over since the BAT will always read 11.4V (using power supply for testing).

On an actual vehicle AUX battery, this would rapidly deplete the charges over a few days...

[jollyjinx]

My first route worked quite well with changing the kernel and going immediately to sleep even before the second core would spin up as it would sometimes reset itself for unknown reasons.
What I finally did is disconnecting 12v on the CAN cable and using a cigarette lighter that provides the 5V USB and will be shut down when the car is sleeping after a few minutes.
As said before I have a BMW i3 which does not automatically charge the 12V battery when its nearing depletion.