ESP32-S3 ESP_PD_DOMAIN_RTC_PERIPH Abnormal power consumption (IDFGH-9205)

[MacWyznawca]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.1-dev-2884-g8a7f6af625

Operating System used.

macOS

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

Development Kit.

custom board with ESP32-S3-WROOM-1 module

Power Supply used.

Battery

What is the expected behavior?

Microamper (uA) power consumption with:
esp_sleep_pd_config(ESP_PD_DOMAIN_RTC_PERIPH, ESP_PD_OPTION_ON); enabled
and runing ULP-RISC-V program ex. ULP-RISC-V ADC Example.

What is the actual behavior?

Consumprion 1.5 mA (miliamper).

When I set rtc_gpio_isolate() for all RTC GPIO power growing to 1.8 mA.

Whit out ESP_PD_DOMAIN_RTC_PERIPH and simple use RTC GPIO (not analog - input and output only) my board use ~100 uA when ULP-RISC-V program run 10 times per second.

Other ESP_IDF ULP-RISC-V examples whit out ESP_PD_DOMAIN_RTC_PERIPH On use ~100 uA too. The problem occurs only when esp_sleep_pd_config(ESP_PD_DOMAIN_RTC_PERIPH, ESP_PD_OPTION_ON) is used.

Steps to reproduce.

Run ULP-RISC-V ADC Example.

Debug Logs.

No response

More Information.

I use ESP32-S3-WROOM-1 module with 4 BM Flash, and no extra PSRAM.

[Veldmonster]

I can confirm. I have the exact same problem also using the ulp_riscv adc example an ESP32-S3-WROOM-1 N8R2 and latest master esp-idf.

[Veldmonster]

I have done a bit more testing. The issue is also present on release 5.0.

However, for v4.4.4 the behavior is different, using the ulp_riscv/adc example:
It starts off by consuming ~1.2mA during sleep but, after waking up once and entering sleep again it consumes ~58uA and continue to do so whenever it enters sleep.

I hope this helps in figuring out what is wrong in v5+.

[MacWyznawca]

Thanks. This good news. Probably is software bug, not in hardware.

[Veldmonster]

Indeed. Also, with the exact same hardware and latest esp-idf-master, using the deep_sleep example it consumes ~25uA in deep sleep.

[Veldmonster]

I have done a few more tests. The latest v5.0 and 5.0.1 exhibit the behavior I described previously for v4.4.4 as in:
"It starts off by consuming ~1.2mA during ULP sleep but, after waking up once and entering sleep again it consumes ~58uA and continue to do so whenever it enters sleep."

It is not ideal that it starts by consuming 1.2mA but, at least it recovers during consecutive sleeps. However, for v5.0 and v5.01 the ULP crashes intermittently. Which is not the case using the latest master.

I believe the following commit is responsible for introducing the regression in master:
adc: improve power logic
If I use an earlier commit, the behavior reverts to the same as <= 5.0.1 as described above.

[johnboiles]

I'm seeing this issue as well, so I've been doing some testing with an as-simple-as-possible ULP example that just does a while (1) {} from the ULP. I'm testing using an Unexpected Maker ProS3 dev board and a Nordic Power Profiling Kit 2 connected to 3.3V.

My findings confirm @Veldmonster's conclusion that 5be3c21 is the problem.

Here are my tests with various versions of IDF:

5be3c21 gives 1.75mA

f9da48d (the commit immediately before 5be3c21) gives 430uA

5.0.1 (a4afa44) gives 417uA

4.4.4 (e8bdaf9) gives 415uA (using a version of my example tweaked to compile with 4.4.4)

[johnboiles]

Also tested with the ULP running every 200ms and doing a for (int i=0; i < 100000; i++) {} (instead of an infinite loop on the ULP). When doing that I don't see the same power discrepancy! Interesting!

5be3c21 25.6uA

f9da48d (the commit immediately before 5be3c21) 25.59uA

5.0.1 (a4afa44) 25.6uA

4.4.4 (e8bdaf9) 25.5uA

[MacWyznawca]

@johnboiles this second measure session is with analog measure in RTC? If not, this is normal. ULP if run all time use ~ 400--600 uA.

[johnboiles]

Ah, but if I explicitly set esp_sleep_pd_config(ESP_PD_DOMAIN_RTC_PERIPH, ESP_PD_OPTION_ON); then I see the discrepancy again between 5be3c21 and the commit immediately before it.

5be3c21 1.36mA

f9da48d (the commit immediately before 5be3c21) 25.59uA

[johnboiles]

@MacWyznawca take a look at my example code in sindarin-inc/minimal-ulp. Im not doing anything with the ADC but still seeing the same behavior with the ULP.

The first test did while(1) {} from the ULP, and the subsequent tests did for (int i=0; i < 100000; i++) {} from the ULP (see the forloop/idf-5) branch. That's all!

[MacWyznawca]

Thank you @johnboiles very much, for the tests and another confirmation that this is not the problem of my system.
That is, the problem is the power-on itself for the RTC peripherals.

It's just a shame that Espressif isn't doing anything about it.

We are about to start serial production, and I can't do analog measurements in sleep using ULP.
Because of this, I have to wake up the whole system unnecessarily.

[johnboiles]

FWIW I'm seeing the same behavior on IDF 5.0.1 stable. But not on f9da48d.

Pulling the latest IDF master (81e1e65) and then moving ulp_set_wakeup_period to be the absolute last command before calling esp_deep_sleep_start seems to give me the expected uA low power ulp. Though I'm not convinced there's not something else at play. I'll keep testing.

[MacWyznawca]

@johnboiles Can you try it on the exampe from IDF with analog measurement in RTC?
I am still waiting for the new prototypes.

[boarchuz]

@johnboiles Does commenting this line fix the problem for you (and cause no other problems)?

esp-idf/components/esp_hw_support/port/esp32s3/sar_periph_ctrl.c

Line 32 in 81e1e65

sar_ctrl_ll_set_power_mode(SAR_CTRL_LL_POWER_ON);

Please take a quick look at the related closed issue. I don't think this is specific to the ULP (at least, it's not for ESP32; I haven't tried with S3).

[johnboiles]

@boarchuz I'll try that out thanks for the suggestion. I need to collect more data but I noticed something like what you're mentioning. I think I noticed the higher power consumption even when not deep sleeping / running the ulp. Harder to see 1.6mA on top the normal active CPU current but but I think I noticed a difference.

[johnboiles]

@boarchuz That does fix it for my trivial sindarin-inc/minimal-ulp example in IDF 5.1 (51772f4) (still testing with the ProS3)! Without your change i'm seeing: 1.36mA and with it I'm seeing 25uA.

Using the same patch on my code seems to be fine. I'm not seeing any issues with it.

Edit: Testing my code on my hardware (a little higher power consumption) the suggested change helps the first sleep but is the same-ish after the first sleep/wake cycle. Interesting that in my code IDF5.0.1 is busted in a similar way (but not on my minimal-ulp example)!

[johnboiles]

Another piece of information. I'm able to get my code running at low power in 5.0.1 if I delete my ADC oneshot unit before sleeping:

ESP_ERROR_CHECK(adc_oneshot_del_unit(adcOneshotHandle));

It does sometimes seem to get stuck at higher power but then it typically drops down then settles somewhere ~225uA (which is roughly expected for our hardware)

Every now and then it will get stuck in deep sleep without dropping down, but it will usually fix itself after the next time the main CPU wakes up.

[johnboiles]

If I initialize the ADC in my sindarin-inc/minimal-ulp example, I'm able to reproduce the high power consumption issue in the following IDF versions:

• IDF 5.0.1
• IDF 5.1.0 (51772f4)
• IDF 5.1.0 (51772f4) w/ the sar_periph_ctrl.c:32 line commented out

In all cases I'm seeing 1.3mA in the ULP. I'm simply adding this code to the top of app_main:

    adc_oneshot_unit_init_cfg_t adcOneshotCfg;
    adc_oneshot_unit_handle_t adcOneshotHandle;
    adcOneshotCfg.unit_id = ADC_UNIT_1;
#if ESP_IDF_VERSION >= ESP_IDF_VERSION_VAL(5, 1, 0)
    adcOneshotCfg.clk_src = TIMER_SRC_CLK_APB;
#endif
    adcOneshotCfg.ulp_mode = ADC_ULP_MODE_DISABLE;
    ESP_ERROR_CHECK(adc_oneshot_new_unit(&adcOneshotCfg, &adcOneshotHandle));

Then of course if I deinit the ADC, it works as expected and I see 25uA again. This might be an easy quick fix for anyone running into this issue since it doesn't require a modified IDF.

ESP_ERROR_CHECK(adc_oneshot_del_unit(adcOneshotHandle));

Not sure if this is expected or not. I would have expected esp_deep_sleep_start to shut down any peripherals that weren't explicitly configured to be used in deep sleep / ulp. There's certainly a bug where the ADC gets powered up even if you're not using it.

[esp-wzh]

Hi, all, we have fixed this problem internally, it will be synced to github soon.

[Veldmonster]

@esp-lis My tests setup:

1. Latest master branch with @esp-wzh suggested fix applied.
2. Test code: ULP-RISC-V ADC Example
3. Battery and Current Ranger to measure the current.

My results indicate that the suggested fix from @esp-wzh does not work.
Side note: I don't see how TSENS state is relevant. The reported issue is specific to ULP, ADC use.

[KonssnoK]

hello @esp-wzh , i think this is related to what i'm seeing on our devices. ESPS3 N8R8 on v4.4 (latest branch 3cec3a0 )

Deep sleep seems to work fine for amount of times below 30 minutes, and to wakeup instead at some point for 30+ minutes.

here a chart of the temperature of the CPU

you can see that for 20 and 30 minutes (which became 26 minutes for unknown reasons) the CPU temperature drops, but for 40 minutes (started at 21:23) we have data (cpu not stopped)

[MacWyznawca]

Please take a quick look at the related closed issue. I don't think this is specific to the ULP (at least, it's not for ESP32; I haven't tried with S3).

Unfortunately, in the case of the ESP32-S3, this did not help. Still after using esp_sleep_pd_config(ESP_PD_DOMAIN_RTC_PERIPH, ESP_PD_OPTION_ON); the current consumption increases from 10 uA to 1.3 mA

[MacWyznawca]

Still after using esp_sleep_pd_config(ESP_PD_DOMAIN_RTC_PERIPH, ESP_PD_OPTION_ON); current consumption increases from 10 uA to 1.3 mA.
Are there any new ideas? We have to go into production soon, and bugs in ESP libraries mna not allow it.

[boarchuz]

Still after using esp_sleep_pd_config(ESP_PD_DOMAIN_RTC_PERIPH, ESP_PD_OPTION_ON); current consumption increases from 10 uA to 1.3 mA. Are there any new ideas? We have to go into production soon, and bugs in ESP libraries mna not allow it.

You can use sar_ctrl_ll_set_power_mode(SAR_CTRL_LL_POWER_FSM);

If ULP ADC isn't working as expected, you may need to manually power the ADC on and off using the ULP as required using SAR_CTRL_LL_POWER_ON and SAR_CTRL_LL_POWER_OFF. Hopefully FSM works.

[MacWyznawca]

@boarchuz Unfortunately, using SAR_CTRL_LL_POWER_FSM alone or in combination with SAR_CTRL_LL_POWER_ON /OFF did not help, and in fact made things worse. The chip in the DEEP SLEEP state consumes 90 uA, but when ADC measurement starts, the consumption jumps to 1.7 mA and the ULP program freezes at ulp_riscv_adc_read_channel().

[esp-wzh]

@MacWyznawca @boarchuz @Veldmonster
I located that this problem may be caused by wrong adc power mode configuration in deepsleep mode, can you pick this fix on latest master or release/v5.1 to confirm if the problem is resolved?

[AxelLin]

@esp-wzh
Please provide patch file instead of png file.

[MacWyznawca]

@esp-wzh I test now with v5.2-dev-193-gdf9310ada2-dirty. The problem persists.
I'm afraid that the changes in adc_oneshot.c have no effect on my problem. ULP uses ulp_riscv_adc_read_channel(), and this direct function uses adc_oneshot_ll_*.

We need a solution to the problem very urgently.

[esp-wzh]

@MacWyznawca The key to this change to fix the abnormal current is to move the sar_periph_ctrl_adc_oneshot_power_acquire that switches the adc power mode out of the adc_oneshot_new_uinit, because it will be called before sleep and will not be released, which leads to a bottom current of 1.xmA during sleep. I tested it with ulp_adc example in IDF and indeed it works.

before:

after:

Can you try this example and share more about the application that the current is not normal?

[MacWyznawca]

@esp-wzh I tested with v5.2-dev-350-g56123c52aa and example ULP-RISC-V ADC Example from the same latest version. The problem occurs when I power on esp_sleep_pd_config(ESP_PD_DOMAIN_RTC_PERIPH, ESP_PD_OPTION_ON);, even without performing the first read of ulp_riscv_adc_read_channel(EXAMPLE_ADC_UNIT, EXAMPLE_ADC_CHANNEL);

I have exactly the same effects as in my project. Current consumption 1.3 - 1.5 mA (depending on the set wake-up frequency.

[esp-wzh]

adc_power_fix_56123.patch.txt
Please try this patch on commit 56123c52aa,I tested it and it does work

[MacWyznawca]

Thanks @esp-wzh this patch works!

[igrr]

Let's keep it open until the fix is merged. The issue will get closed automatically when the commit with the fix lands into master branch here on Github.

[patrickmagee]

Hi, thanks @ESP-Marius for pointing me to this thread, I had missed on search.

I applied the above patch to IDF version v5.2-dev-503-g17451f1fb3.
Running the https://github.com/espressif/esp-idf/tree/master/examples/system/ulp_riscv/adc on custom hardware the patch takes my current consumption from ~2mA to ~800uA. Looking at the current profile it's pretty stable, I would have expected some pulse every 20mS using the example wake up :

ulp_set_wakeup_period(0, 200000);

Given the example code, I would expect to see a few 100s of uA for the period the ADC reading is happening, then some much lower current (few 10s of uA ) when the ULP halts and waits for the next firing of its wakeup timer?

int main (void)
{
    int32_t last_result = ulp_riscv_adc_read_channel(EXAMPLE_ADC_UNIT, EXAMPLE_ADC_CHANNEL);

    if (last_result > adc_threshold) {
        wakeup_result = last_result;
        ulp_riscv_wakeup_main_processor();
    }

    return 0;
}

One theory I had was that the ADC was taking too long to complete and over running the 20mS timer, but changing the time even to 500mS makes no difference:

ulp_set_wakeup_period(0, 500000);

Running @johnboiles very useful minimum ulp I get ~34uA (which is expected as I have some peripherals). So there's still something strange with the ADC? I'll build a bare bones hardware to completely rule out any weird i/o issues on my pcb and report back.

[esp-wzh]

@patrickmagee I re-verified the ulp_adc example after applying the patch on the commit point you mentioned, and the measured base current is about 17uA. I think the abnormal current may be caused by some IO leakage on your board.

[patrickmagee]

Thanks @esp-wzh , I figured a few i/o issues that were leaking current but was still seeing ~400uA which didn't seem right.
I'm not exactly sure why I was seeing such a different sleep current to your experiments, even on a bare board build. So I pulled 54576b7 (master) and applied your patch there.

Success! I'm now down to 70uA which is what I would roughly expect with the peripherals I have on the board.

[Veldmonster]

Thanks @esp-wzh, I can also confirm that the patch works. On my board with ulp_set_wakeup_period at 100ms and ulp execution time of 2ms I measure a consumption of ~90uA RMS (~67uA AVG).

[KonssnoK]

when is this going to be backported to 4.4?

[patrickmagee]

I don't speak for espressif but I don't think they backport. Your best option is to update your espidf to the master.

[KonssnoK]

@patrickmagee
this is a fix, not a new feature.
v4.4 is supported and in maintenance till 31 July 2024.
-> it has to be ported.

As a company we cannot change branch that easily (changes from 4.4 to 5.X are extensive)

[patrickmagee]

Ah , apologies for my error, you are correct given that it is a fix.