scripts/diffconfig.sh: cannot parse config.buildinfo from buildbot

[graysky2]

Something is causing scripts/diffconfig.sh to fail. You can reproduce this by getting the build bot config and running the script on it.

wget https://downloads.openwrt.org/snapshots/targets/bcm27xx/bcm2711/config.buildinfo -O .config
make nconfig
 << save the file >>
% ./scripts/diffconfig.sh > ok
Config.in:9: syntax error
Config.in:8: invalid statement
Config.in:9: invalid statement
Config.in:10: invalid statement
can't open file 'tmp/.diffconfig.stage1' at ./scripts/kconfig.pl line 32.
Config.in:9: syntax error
Config.in:8: invalid statement
Config.in:9: invalid statement
Config.in:10: invalid statement
can't open file 'tmp/.diffconfig.stage2' at ./scripts/kconfig.pl line 32.

[jow-]

Does the issue persist after make -C scripts/config/ clean ? I was briefly able to reproduce it but now not anymore.

[Borromini]

@graysky2 I bumped into the same issue earlier but the command jow suggests indeed takes care of it. Might have helped if the commit message for 009293c had the clean command as a suggestion. Because that's what's at the root of the messages you're seeing.

[graysky2]

Ah, so that is an extra step indeed. It is also noisy/verbose. Is that to be expected?

% make -C scripts/config/ clean
make: Entering directory '/scratch/union/scripts/config'
rm -f *.o lxdialog/*.o *.moc .*.cmd zconf.tab.c zconf.lex.c zconf.hash.c .tmp_qtcheck *conf-cfg qconf-moc.cc conf nconf mconf qconf
make: Leaving directory '/scratch/union/scripts/config'

And

% ./scripts/diffconfig.sh > ok
./scripts/diffconfig.sh: line 11: ./scripts/config/conf: No such file or directory
can't open file 'tmp/.diffconfig.stage1' at ./scripts/kconfig.pl line 32.
./scripts/diffconfig.sh: line 13: ./scripts/config/conf: No such file or directory
can't open file 'tmp/.diffconfig.stage2' at ./scripts/kconfig.pl line 32.

Despite the "no such file" output, it did create ok

% cat ok
CONFIG_TARGET_bcm27xx=y
CONFIG_TARGET_bcm27xx_bcm2711=y
CONFIG_TARGET_MULTI_PROFILE=y
CONFIG_TARGET_DEVICE_bcm27xx_bcm2711_DEVICE_rpi-4=y
CONFIG_TARGET_DEVICE_PACKAGES_bcm27xx_bcm2711_DEVICE_rpi-4=""
CONFIG_ALL_KMODS=y
CONFIG_ALL_NONSHARED=y
CONFIG_DEVEL=y
CONFIG_TARGET_PER_DEVICE_ROOTFS=y

[jow-]

The scripts/config/ clean is only required once. Afterwards you should run one of the config targets, such as make defconfig, make menuconfig, make nconfig etc. to trigger a rebuild of the kconfig interpreter. Afterwards diffconfig.sh should function without the No such file or directory error.

[Borromini]

I reported this on #openwrt-devel yesterday, but looking at the logs other people ran into it too, so we're definitely not the only ones.

E.g. my report on IRC and a later one.

[ynezz]

What about following fix? @cotequeiroz @aparcar

diff --git a/include/toplevel.mk b/include/toplevel.mk
index 1e543ebcb98f..ce744bc92ccf 100644
--- a/include/toplevel.mk
+++ b/include/toplevel.mk
@@ -101,7 +101,7 @@ ifneq ($(DISTRO_PKG_CONFIG),)
 scripts/config/%onf: export PATH:=$(dir $(DISTRO_PKG_CONFIG)):$(PATH)
 endif
 scripts/config/%onf: CFLAGS+= -O2
-scripts/config/%onf:
+scripts/config/%onf: FORCE
 	@$(_SINGLE)$(SUBMAKE) $(if $(findstring s,$(OPENWRT_VERBOSE)),,-s) \
 		-C scripts/config $(notdir $@)

Seems to work as expected:

$ rm .config; touch scripts/config/conf.c; make defconfig V=s 2>&1 | grep ^cc
cc -O2   -c -o conf.o conf.c
cc   conf.o confdata.o expr.o lexer.lex.o parser.tab.o preprocess.o symbol.o util.o   -o conf

$ rm .config; make defconfig V=s 2>&1 | grep ^cc

Reproducer is reverting and re-applying of 009293c.

[aparcar]

@ynezz works for me, if it doesn't slow down things, let's do it.

[cotequeiroz]

This was transient, and occurred after of the scrips/config update when you already have the old version built. make does not detect the change, so it will not rebuild the files. The syntax changed between versions, hence the syntax errors.

If you start from a clean clone, you don't get the error. @Borromini is right that a warning about this situation should have been made--or better yet, fixed it--and I apologize for the inconvenience.

@ynezz's fix works in the following case:

$ git clone https://git.openwrt.org/openwrt/openwrt.git .
$ git checkout 73ea763c0dcdb96c9163790f20edd964399035c0 # just before the scrits/config update
$ make defconfig V=scp
... compiles compiles scripts/config and finishes OK
make[1]: Entering directory '/tmp/openwrt/scripts/config'
cc -O2   -c -o conf.o conf.c
cc -O2   -c -o confdata.o confdata.c
cc -O2   -c -o expr.o expr.c
cc -O2 -I ./.   -c -o lexer.lex.o lexer.lex.c
cc -O2 -I ./.   -c -o parser.tab.o parser.tab.c
cc -O2   -c -o preprocess.o preprocess.c
cc -O2   -c -o symbol.o symbol.c
cc -O2   -c -o util.o util.c
cc   conf.o confdata.o expr.o lexer.lex.o parser.tab.o preprocess.o symbol.o util.o   -o conf
make[1]: Leaving directory '/tmp/openwrt/scripts/config'
...
$ git checkout master && make defconfig V=scp
...
[ -L .config ] && export KCONFIG_OVERWRITECONFIG=1; \
       	scripts/config/conf  --defconfig=.config Config.in
Config.in:9: syntax error
Config.in:8: invalid statement
Config.in:9: invalid statement
Config.in:10: invalid statement
make: *** [/home/equeiroz/src/tmp/openwrt/include/toplevel.mk:120: defconfig] Error 1
$ patch -p1 < ../ynezz.patch # apply ynezz's patch
...
$ make defconfig V=scp
make[1]: Entering directory '/tmp/openwrt/scripts/config'
cc -O2    -c -o conf.o conf.c
cc -O2    -c -o confdata.o confdata.c
cc -O2 -I ./.   -c -o lexer.lex.o lexer.lex.c
cc -O2    -c -o menu.o menu.c
cc -O2 -I ./.   -c -o parser.tab.o parser.tab.c
cc -O2    -c -o preprocess.o preprocess.c
cc -O2    -c -o symbol.o symbol.c
cc   conf.o confdata.o expr.o lexer.lex.o menu.o parser.tab.o preprocess.o symbol.o util.o   -o conf
make[1]: Leaving directory '/tmp/openwrt/scripts/config'
... finishes with no error.
$ make defconfig V=scp
make[1]: Entering directory '/tmp/openwrt/scripts/config'
make[1]: 'conf' is up to date.
make[1]: Leaving directory '/tmp/openwrt/scripts/config'
... does not rebuild scrips/config

However, there's still a glitch that needs to be fixed. If instead of make defconfig, as I did in the example above, one runs make nconfig, or make menuconfig (but not make defconfig at any time), then scripts/config/conf does not get built, and ./scritps/diffconfig.sh produces a diffconfig, but shows these errors:

./scripts/diffconfig.sh: line 11: ./scripts/config/conf: No such file or directory
can't open file 'tmp/.diffconfig.stage1' at ./scripts/kconfig.pl line 32.
./scripts/diffconfig.sh: line 13: ./scripts/config/conf: No such file or directory
can't open file 'tmp/.diffconfig.stage2' at ./scripts/kconfig.pl line 32.

This was happening before 009293c, but has not been caught until now.

$ git clone https://git.openwrt.org/openwrt/openwrt.git .; git checkout 73ea763c0dcdb96c9163790f20edd964399035c0; make nconfig

Select libopenssl and save. You can use menuconfig, xconfig, nconfig, above. Any of them will fail next:

$ ./scripts/diffconfig.sh
./scripts/diffconfig.sh: line 11: ./scripts/config/conf: No such file or directory
can't open file 'tmp/.diffconfig.stage1' at ./scripts/kconfig.pl line 32.
./scripts/diffconfig.sh: line 13: ./scripts/config/conf: No such file or directory
can't open file 'tmp/.diffconfig.stage2' at ./scripts/kconfig.pl line 32.
CONFIG_TARGET_ath79=y
CONFIG_TARGET_ath79_generic=y
CONFIG_TARGET_ath79_generic_DEVICE_8dev_carambola2=y

Notice the absence of CONFIG_PACKAGE_libopenssl=y. Inserting make -s ./scripts/config/conf before the first call to it in scripts/diffconfig.sh will fix it:

--- a/scripts/diffconfig.sh
+++ b/scripts/diffconfig.sh
@@ -8,6 +8,7 @@ grep '^CONFIG_DEVEL=y' .config >> tmp/.diffconfig.head
 grep '^CONFIG_TOOLCHAINOPTS=y' .config >> tmp/.diffconfig.head
 grep '^CONFIG_BUSYBOX_CUSTOM=y' .config >> tmp/.diffconfig.head
 grep '^CONFIG_TARGET_PER_DEVICE_ROOTFS=y' .config >> tmp/.diffconfig.head
+make -s ./scripts/config/conf
 ./scripts/config/conf --defconfig=tmp/.diffconfig.head -w tmp/.diffconfig.stage1 Config.in >/dev/null
 ./scripts/kconfig.pl '>+' tmp/.diffconfig.stage1 .config >> tmp/.diffconfig.head
 ./scripts/config/conf --defconfig=tmp/.diffconfig.head -w tmp/.diffconfig.stage2 Config.in >/dev/null

Edit: to be clear: The fix needs both patches (@ynezz's and mine) to work.

[cotequeiroz]

@ynezz, if you want to push the fix to diffconfig.sh along with yours, here's the commit:
cotequeiroz/openwrt@01a3242

[ynezz]

@ynezz, if you want to push the fix to diffconfig.sh along with yours, here's the commit: cotequeiroz/openwrt@01a3242

@cotequeiroz Thanks!

[ynezz]

@cotequeiroz seems like c0849c1 has likely introduced regression with config.buildinfo as it has wrong content now:

curl -s https://downloads.openwrt.org/snapshots/targets/ath79/generic/config.buildinfo | head -2
make[3]: Entering directory '/builder/shared-workdir/build'
make[3]: Leaving directory '/builder/shared-workdir/build'

[cotequeiroz]

Yeah, old versions of make apparently add w to MAKEFLAGS, despite the use of -s flag when calling the submake.

I have a few ideas about how to proceed:

1. make ./scripts/config/conf >/dev/null || { make ./scripts/config/conf; exit 1 }: showing errors on the second call does not guarantee that they'll be the same as in the original run. It may be able to run through transient problems as well, which may or may not be desirable.
2. I don't like this one so much because of the penalty of writing to a temporary file, but it will produce true output:

make ./scripts/config/conf > ./tmp/diffconfig-make-conf.out || { echo ./tmp/diffconfig-make-conf.out; rm -f ./tmp/diffconfig-make-conf.out; exit 1; }; rm ./tmp/diffconfig-make-conf.out

3. plain make >/dev/null || exit 1 will ensure that nothing will bother diffconfig.sh, while making debugging harder.
4. return EWONTFIX. make diffconfig already builds conf, so it does not need any change.

Notice that I'm aborting if make fails. I don't know if this is desirable or not.

I would rather not proceed than risking to write a bad file that is apparently OK, which is a possible scenario of passing new Config.in data to an old version of conf that was lurking around when make failed.

Aborting at the stage where I had the make command in the previous commit will still generate a temp file, so I think it is best to run make first. I'll do this in a new PR shortly.

[ynezz]

Should be fixed, thanks.