sdkconfig changes when project is built on different machines (IDFGH-12478)

[krupis]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

5.0.4

Operating System used.

Windows

How did you build your project?

VS Code IDE

If you are using Windows, please specify command line type.

PowerShell

What is the expected behavior?

Build the project without any sdkconfig conflicts

What is the actual behavior?

We have multiple people working on the same esp-idf project. We have noticed that whenever someone makes changes and push to the repository, whenever other people try to pull all the changes and build the project on their own machine, during build process something changes on the sdkconfig.h.

I want to understand why does the sdkconfig.h changes when project is built on different machines even if the same esp-idf version is used?

Steps to reproduce.

1. Pull someones changes from the repository
2. Build project

Build or installation Logs.

For example, I have just pulled changes from my colleague and built the project. I can see that changes are automatically detected using the Source Control extension on the VSCode on the sdkconfig.

It seems that the only changes are the order. Please see the example below:

As you can see from 2 images above, on my machine, LittleFS is configured starting on the line 1675 and on my colleagues sdkconfig it is configured starting at line 1737. What is the reason for that and why that happens?

More Information.

No response

[dobairoland]

Because items before writing those files are not sorted and their order is semi-random and could be different between systems just as you observe.

You are not supposed to commit sdkconfig or anything from the build directory, including sdkconfig.h, into source control management systems. Any changes different from the default ones should be stored in sdkconfig.defaults and that could be committed. See idf.py save-defconfig for how to generate that file. You can add items to that file manually as well. sdkconfig should be removed each time there is a change in sdkconfig.defaults or the changes will be ignored.

[dobairoland]

I was notified that we should support committing sdkconfig files into source control systems, and therefore, deterministic order in that file. So I'm reopening this....

[krupis]

So everytime I make a change in sdkconfig asuming I use menuconfig what is being changed is the sdkconfig right? So when I am ready to upload new changes to the git, I have to manually copy and paste all the changes and put them to sdkconfig.defaults? That sounds like unnecessary extra step

Perhaps I do not fully understand how does sdkconfig and sdkconfig.defaults work yet. Im reading about it as we speak.

I simply want to understand what is the optimal way of making changes in the menuconfig and pushing those changes to the repository while avoiding the issue I have described above.

[KaeLL]

So everytime I make a change in sdkconfig asuming I use menuconfig what is being changed is the sdkconfig right?

Yes

So when I am ready to upload new changes to the git, I have to manually copy and paste all the changes and put them to sdkconfig.defaults?

No. You can just run idf.py save-defconfig.
Of if you haven't opened menuconfig yet, you can delete the sdkconfig file, open menuconfig, make the changes you want, press D, and save it to ../sdkconfig.defaults.

[krupis]

So everytime I make a change in sdkconfig asuming I use menuconfig what is being changed is the sdkconfig right?

Yes

So when I am ready to upload new changes to the git, I have to manually copy and paste all the changes and put them to sdkconfig.defaults?

No. You can just run idf.py save-defconfig.
Of if you haven't opened menuconfig yet, you can delete the sdkconfig file, open menuconfig, make the changes you want, press D, and save it to ../sdkconfig.defaults.

Could you just confirm what is the purpose of that and why cant we just simply commit sdkconfig instead?

[KaeLL]

You can commit it, but the file is unstable given that configs are added/removed/moved/renamed at will, resulting in a lot of noise each time IDF is updated and a Kconfig is touched.

[dobairoland]

Hi @krupis.

There is a line in tools/cmake/project.cmake at https://github.com/espressif/esp-idf/blob/v5.0.4/tools/cmake/project.cmake#L292:

file(GLOB component_dirs ${component_dir}/*)

Could you please try adding the following line after it?

list(SORT component_dirs)

This has to be done on both computers where you experience the difference in the sdkconfig content.

Could you please try this and report back if this fixed the issue? I'm just guessing because I cannot reproduce it.

[mickeyl]

@dobairoland

I have a similar problem, but more annoying in so far that it happens on the same machine.

A call of idf.py build seems to reorder certain parts of the sdkconfig, so that I get a bogus diff afterwards.
Adding the suggested list(SORT component_dirs) in project.cmake does not make a difference for me.

Anything else I could try?

[dobairoland]

The greatest help would be to guide us to reproduce it.

Which parts of sdkconfig are changing? There is a one block which is moving around or there are more moving parts? It is always completely different or there is a pattern?

Where the moving parts are coming from? Is it a component in ESP-IDF, added by an environment variable, a local component in the project directory, obtained by component manager, ....?

[mickeyl]

Complete blocks are moving, always the same blocks. Those blocks are coming in from Kconfig files that are part of components referenced via (local) dependencies specified in main/idf_components.yml.

[ConnorDuvalS]

I believe @mickeyl is right about #10419 being related to this. @dobairoland there is a full step-by-step reproduction guide in my comment at #10419 (comment).

[dobairoland]

Let us wait for espressif/idf-component-manager#58 to see if the ordering can be fixed on the component manager side.

2f811b7 already addressed ordering on the ESP-IDF side.