SPIFFS可以成功挂载文件系统，POSSIX / C均可以打开文件，但是读文件失败，返SPIFFS_ERR_INDEX_REF_LU [-10016] (GIT8266O-328) #766

EaonL · 2019-11-24T07:42:17Z

环境描述：
ESP8266 + 4M Bytes SPI FLASH 【W25Q32】+ SDK release/v3.2

分区定义：
nvs, data, nvs, 0x9000, 0x6000,
phy_init, data, phy, 0xf000, 0x1000,
factory, app, factory, 0x10000, 512K,
storage, data, spiffs, , 448K,

操作步骤：

使用esptools.py工具先整体擦除Flash；
创建spiffs文件系统镜像目录：
mkdir spiffs_root
vim hello.txt 【随便写了一串字符串在文件里】
使用如下命令制作spiffs文件系统镜像：
python ./tools/spiffsgen.py 0x70000 spiffs_root/ spiffs.bin
在storage分区烧写文件系统镜像：
python ./tools/esptool.py --chip esp8266 --port /dev/ttyUSB0 --baud 115200 write_flash -z 0x90000 spiffs.bin
改写SPIFFS_CHECK_RES宏函数，打印相关错误信息：
#define SPIFFS_CHECK_RES(res)
do {
if ((res) < SPIFFS_OK)
{
printf("Err!! file: %s, func: %s, line: %d, ErrNo: %d\n\n", FILE, func, LINE, res);
return (res);
}
} while (0);
make menuconfig将Log out等级设置为Debug模式，并打开SPIFFS的所有输出；
改写example/storage/spiffs/main/spiffs_example_main.c :
7.1 挂载spiffs；
7.2 POSSIX 或 C 编写open("/spiffs/hello.txt"，...)，read(fd, ...)；
make flash && make monitor；
输出如下日志：

D (701) SPIFFS: read: load objix page 0001:0000 for data spix:0000

D (711) SPIFFS: read: offset:0 rd:20 data spix:0000 is data_pix:ff00 addr:00ff0005

Err!! file: /root/esp/ESP8266_RTOS_SDK/components/spiffs/spiffs/src/spiffs_nucleus.c, func: spiffs_object_read, line: 2036, ErrNo: -10016

D (741) SPIFFS: SPIFFS_clearerr

Note:
spiffs的问题库上找到一条类似的问题描述： pellepl/spiffs#160
他的环境是MSP430 + IAR，由于地址对齐问题引发，他使能cache可以read；然而我在ESP8266上各种折腾也不能成功读出文件，我有尝试使用esptools.py工具读Flash，读出来的内容和写进去的内容一致。

EaonL · 2019-11-24T09:28:59Z

问题已解决：
问题原因：引发这个问题的根本原因应该是spiffsgen.py工具配置的SPIFFS_ALIGNED_OBJECT_INDEX_TABLES与SDK配置的不一致，SDK将SPIFFS_ALIGNED_OBJECT_INDEX_TABLES配置为4，不清楚spiffsgen.py工具配置的值是多少。
解决方法：使用mkspiffs制作文件系统镜像，编译mkspiffs时将SPIFFS宏配置与SDK中的值设置成一致。

git clone https://github.com/igrr/mkspiffs.git
cd mkspiffs
git submodule update --init
make dist CPPFLAGS=“-DSPIFFS_OBJ_META_LEN=4 -DSPIFFS_ALIGNED_OBJECT_INDEX_TABLES=4" BUILD_CONFIG_NAME=”-esp-8266”
./tools/mkspiffs/mkspiffs -c spiffs_root_win/ -b 4096 -p 256 -s 0x70000 spiffs.bin
python ./tools/esptool.py --chip esp8266 --port /dev/ttyUSB0 --baud 115200 write_flash -z 0x90000 spiffs.bin
make flash && make monitor
成功打开文件并读取内容。

EaonL · 2019-11-26T02:11:16Z

spiffsgen.py生成的spiffs.bin文件系统镜像和SDK配置的spiffs格式不一致，使用mkspiffs工具将spiffs文件系统镜像的格式与SDK配成一致，问题解决。

tius2000 · 2019-11-26T10:09:32Z

Hi @EaonL,
thank you for posting your results! I have the same problem and still get an error SPIFFS_ERR_INDEX_REF_LU [-10016]:

$ grep SPIFFS <sdkconfig
CONFIG_USING_SPIFFS=y
CONFIG_SPIFFS_MAX_PARTITIONS=3
CONFIG_SPIFFS_CACHE=y
CONFIG_SPIFFS_CACHE_WR=y
# CONFIG_SPIFFS_CACHE_STATS is not set
CONFIG_SPIFFS_PAGE_CHECK=y
CONFIG_SPIFFS_GC_MAX_RUNS=10
# CONFIG_SPIFFS_GC_STATS is not set
CONFIG_SPIFFS_PAGE_SIZE=256
CONFIG_SPIFFS_OBJ_NAME_LEN=32
CONFIG_SPIFFS_USE_MAGIC=y
CONFIG_SPIFFS_USE_MAGIC_LENGTH=y
CONFIG_SPIFFS_META_LENGTH=4
CONFIG_SPIFFS_USE_MTIME=y
# CONFIG_SPIFFS_DBG is not set
# CONFIG_SPIFFS_API_DBG is not set
# CONFIG_SPIFFS_GC_DBG is not set
# CONFIG_SPIFFS_CACHE_DBG is not set
# CONFIG_SPIFFS_CHECK_DBG is not set
# CONFIG_SPIFFS_TEST_VISUALISATION is not set

$ mkspiffs --version
mkspiffs ver. 0.2.3-6-g983970e
Build configuration name: esp-8266
SPIFFS ver. 0.3.7-5-gf5e26c4
Extra build flags: -DSPIFFS_OBJ_META_LEN=4 -DSPIFFS_ALIGNED_OBJECT_INDEX_TABLES=4
SPIFFS configuration:
  SPIFFS_OBJ_NAME_LEN: 32
  SPIFFS_OBJ_META_LEN: 4
  SPIFFS_USE_MAGIC: 1
  SPIFFS_USE_MAGIC_LENGTH: 1
  SPIFFS_ALIGNED_OBJECT_INDEX_TABLES: 4

Is there a wrong setting? Could you please post the relevant settings of your sdkconfig?

EaonL · 2019-11-26T10:57:41Z

Hi @EaonL,
thank you for posting your results! I have the same problem and still get an error SPIFFS_ERR_INDEX_REF_LU [-10016]:

$ grep SPIFFS <sdkconfig
CONFIG_USING_SPIFFS=y
CONFIG_SPIFFS_MAX_PARTITIONS=3
CONFIG_SPIFFS_CACHE=y
CONFIG_SPIFFS_CACHE_WR=y
# CONFIG_SPIFFS_CACHE_STATS is not set
CONFIG_SPIFFS_PAGE_CHECK=y
CONFIG_SPIFFS_GC_MAX_RUNS=10
# CONFIG_SPIFFS_GC_STATS is not set
CONFIG_SPIFFS_PAGE_SIZE=256
CONFIG_SPIFFS_OBJ_NAME_LEN=32
CONFIG_SPIFFS_USE_MAGIC=y
CONFIG_SPIFFS_USE_MAGIC_LENGTH=y
CONFIG_SPIFFS_META_LENGTH=4
CONFIG_SPIFFS_USE_MTIME=y
# CONFIG_SPIFFS_DBG is not set
# CONFIG_SPIFFS_API_DBG is not set
# CONFIG_SPIFFS_GC_DBG is not set
# CONFIG_SPIFFS_CACHE_DBG is not set
# CONFIG_SPIFFS_CHECK_DBG is not set
# CONFIG_SPIFFS_TEST_VISUALISATION is not set

$ mkspiffs --version
mkspiffs ver. 0.2.3-6-g983970e
Build configuration name: esp-8266
SPIFFS ver. 0.3.7-5-gf5e26c4
Extra build flags: -DSPIFFS_OBJ_META_LEN=4 -DSPIFFS_ALIGNED_OBJECT_INDEX_TABLES=4
SPIFFS configuration:
  SPIFFS_OBJ_NAME_LEN: 32
  SPIFFS_OBJ_META_LEN: 4
  SPIFFS_USE_MAGIC: 1
  SPIFFS_USE_MAGIC_LENGTH: 1
  SPIFFS_ALIGNED_OBJECT_INDEX_TABLES: 4

Is there a wrong setting? Could you please post the relevant settings of your sdkconfig?

This is my config:

./mkspiffs --version
mkspiffs ver. 0.2.3-6-g983970e
Build configuration name: EaonConfig_ESP8266
SPIFFS ver. 0.3.7-5-gf5e26c4
Extra build flags: -DSPIFFS_OBJ_META_LEN=4 -DSPIFFS_ALIGNED_OBJECT_INDEX_TABLES=4
SPIFFS configuration:
SPIFFS_OBJ_NAME_LEN: 32
SPIFFS_OBJ_META_LEN: 4
SPIFFS_USE_MAGIC: 1
SPIFFS_USE_MAGIC_LENGTH: 1
SPIFFS_ALIGNED_OBJECT_INDEX_TABLES: 4

grep SPIFFS sdkconfig
CONFIG_USING_SPIFFS=y
CONFIG_SPIFFS_MAX_PARTITIONS=3
# SPIFFS Cache Configuration
CONFIG_SPIFFS_CACHE=y
CONFIG_SPIFFS_CACHE_WR=y
CONFIG_SPIFFS_CACHE_STATS=
CONFIG_SPIFFS_PAGE_CHECK=y
CONFIG_SPIFFS_GC_MAX_RUNS=10
CONFIG_SPIFFS_GC_STATS=
CONFIG_SPIFFS_PAGE_SIZE=256
CONFIG_SPIFFS_OBJ_NAME_LEN=32
CONFIG_SPIFFS_USE_MAGIC=y
CONFIG_SPIFFS_USE_MAGIC_LENGTH=y
CONFIG_SPIFFS_META_LENGTH=4
CONFIG_SPIFFS_USE_MTIME=y
CONFIG_SPIFFS_DBG=
CONFIG_SPIFFS_API_DBG=
CONFIG_SPIFFS_GC_DBG=
CONFIG_SPIFFS_CACHE_DBG=
CONFIG_SPIFFS_CHECK_DBG=
CONFIG_SPIFFS_TEST_VISUALISATION=

Which version of the SDK are you using？

tius2000 · 2019-11-26T12:53:58Z

Dear @EaonL,

This is my config:
...

Thank you for our config. I can see no relevant differences.

Which version of the SDK are you using？

sdk version v3.2-261-g7ab9602e-dirty

I have no idea what is wrong :-(

EaonL · 2019-11-26T14:28:23Z

Dear @EaonL,

This is my config:
...

Thank you for our config. I can see no relevant differences.

Which version of the SDK are you using？

sdk version v3.2-261-g7ab9602e-dirty

I have no idea what is wrong :-(

This question is annoying，but it will definitely be resolved in the end.
I see your command line output <sdkconfig, what version of Linux do you use? maybe u should try to use Ubuntu 16.04 TLS.

Beside, the mkffs tool is generated with your system's gcc compiler，check the version of your system's gcc compiler. xtensa-lx106-elf-gcc is a cross-link provided by Espressif, if your system's gcc version is not consistent with the gcc version of the compiled xtensa-lx106-elf-gcc cross toolchain, it may also cause the file to be unreadable:

root@ubuntu:~# gcc --version
gcc (GCC) 5.2.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

root@ubuntu:~# xtensa-lx106-elf-gcc --version
xtensa-lx106-elf-gcc (crosstool-NG crosstool-ng-1.22.0-100-ge567ec7) 5.2.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

See, the gcc version of my system is the same as the gcc version that generated xtensa-lx106-elf-gcc, they are both Version-5.2.0.

At last, I provide you a file system image 《spiffs_img.bin》, you can burn this file system image and read the file 《eaon.txt》 to see if it can be read successfully:

root@ubuntu:/root/spiffs_img# pwd
/root/spiffs_img
root@ubuntu:/root/spiffs_img#

root@ubuntu:/root/spiffs_img# ls
eaon.txt
root@ubuntu:/root/spiffs_img#

root@ubuntu:/root/spiffs_img# cat eaon.txt
Hello! This is a file from eaon!!
[ Note, in file eaon.txt, here is a '\n' ]
root@ubuntu:/root/spiffs_img#

root@ubuntu:/root/spiffs_img# cd ..
root@ubuntu:/root#

root@ubuntu:/root# ./tools/mkspiffs/mkspiffs -c spiffs_img/ -b 4096 -p 256 -s 0x70000 spiffs_img.bin

root@ubuntu:/root# stat spiffs_img.bin
File: 'spiffs_img.bin'
Size: 458752 Blocks: 896 IO Block: 4096 regular file
Device: 801h/2049d Inode: 527112 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2019-11-26 05:47:31.723173135 -0800
Modify: 2019-11-26 05:47:31.723173135 -0800
Change: 2019-11-26 05:47:31.723173135 -0800
Birth: -
root@ubuntu:/root#

As you can see, the size of spiffs_img.bin was defined as 0x70000, which means you need to set the partition of spiffs to 448K.

root@ubuntu:/root# cd $IDF_PATH/examples/storage/spiffs/

root@ubuntu:/root/esp/ESP8266_RTOS_SDK/examples/storage/spiffs# cat partitions_example.csv
#Name, Type, SubType, Offset, Size, Flags
#Note: if you change the phy_init or app partition offset, make sure to change the offset in Kconfig.projbuild
nvs, data, nvs, 0x9000, 0x6000,
phy_init, data, phy, 0xf000, 0x1000,
factory, app, factory, 0x10000, 512K,
storage, data, spiffs, 0x90000, 448K,
root@ubuntu:/root/esp/ESP8266_RTOS_SDK/examples/storage/spiffs#

python /root/tools/esptool.py --chip esp8266 --port /dev/ttyUSB0 --baud 115200 write_flash -z 0x90000 spiffs_img.bin

modify main/spiffs_example_main.c to read /spiffs_img/eaon.txt:
//The default mount in the Espressif-Example is fine
......
// Open file for reading
ESP_LOGI(TAG, "Reading file");
FILE* f = fopen("/spiffs/eaon.txt", "r");
if (f == NULL) {
ESP_LOGE(TAG, "Failed to open file for reading");
return;
}
char line[64];
fgets(line, sizeof(line), f);
fclose(f);
// strip newline
char* pos = strchr(line, '\n');
if (pos) {
*pos = '\0';
}
ESP_LOGI(TAG, "Read from file /spiffs_img/eaon.txt: '%s'", line);

make flash && make monitor

Then, hope u can see, good luck:
I (497) reset_reason: RTC reset 2 wakeup 0 store 0, reason is 2
I (501) example: Initializing SPIFFS
I (533) example: Partition size: total: 414401, used: 502
I (536) example: Reading file
I (539) example: Read from file /spiffs_img/eaon.txt: 'Hello! This is a file from eaon!!'

You should unzip the 《spiffs_img.zip》, and then you get 《spiffs_img.bin》.
spiffs_img.zip

tius2000 · 2019-11-27T17:11:52Z

I see your command line output <sdkconfig, what version of Linux do you use? maybe u should try to use Ubuntu 16.04 TLS.

I'm using Windows 10 with the msys32 enviroment provided by EspressIF.

Beside, the mkffs tool is generated with your system's gcc compiler，check the version of your system's gcc compiler. xtensa-lx106-elf-gcc is a cross-link provided by Espressif, if your system's gcc version is not consistent with the gcc version of the compiled xtensa-lx106-elf-gcc cross toolchain, it may also cause the file to be unreadable:

I use the gcc versions provided by EpressIF within the msys32 enviroment:

$ gcc --version
gcc (GCC) 7.3.0
...
$ which gcc.exe
/usr/bin/gcc.exe
$  xtensa-lx106-elf-gcc --version
xtensa-lx106-elf-gcc.exe (crosstool-NG crosstool-ng-1.22.0-100-ge567ec7b) 5.2.0
...

You are right, there is a mismatch!

At last, I provide you a file system image 《spiffs_img.bin》, you can burn this file system image and read the file 《eaon.txt》 to see if it can be read successfully:

Your file is working :-)

file /spiffs/eaon.txt:
Hello! This is a file from eaon!!

So neither spiffsgen.py nor mkspiffs produce a valid image on my windows system. Perhaps this is really caused by the different gcc versions.

To depend on specific compiler versions seems a bit difficult. Perhaps it is possible to remove this dependency by adding esp8266 support to spiffsgen.py?

EaonL · 2019-11-28T03:48:49Z

I see your command line output <sdkconfig, what version of Linux do you use? maybe u should try to use Ubuntu 16.04 TLS.

I'm using Windows 10 with the msys32 enviroment provided by EspressIF.

Beside, the mkffs tool is generated with your system's gcc compiler，check the version of your system's gcc compiler. xtensa-lx106-elf-gcc is a cross-link provided by Espressif, if your system's gcc version is not consistent with the gcc version of the compiled xtensa-lx106-elf-gcc cross toolchain, it may also cause the file to be unreadable:

I use the gcc versions provided by EpressIF within the msys32 enviroment:
$ gcc --version
gcc (GCC) 7.3.0
...
$ which gcc.exe
/usr/bin/gcc.exe
$  xtensa-lx106-elf-gcc --version
xtensa-lx106-elf-gcc.exe (crosstool-NG crosstool-ng-1.22.0-100-ge567ec7b) 5.2.0
...
You are right, there is a mismatch!

At last, I provide you a file system image 《spiffs_img.bin》, you can burn this file system image and read the file 《eaon.txt》 to see if it can be read successfully:

Your file is working :-)
file /spiffs/eaon.txt:
Hello! This is a file from eaon!!
So neither spiffsgen.py nor mkspiffs produce a valid image on my windows system. Perhaps this is really caused by the different gcc versions.

To depend on specific compiler versions seems a bit difficult. Perhaps it is possible to remove this dependency by adding esp8266 support to spiffsgen.py?

Maybe you should use your system's gcc to compile the toolchain yourself, rather than using the toolchain binary package compiled by Espressif. I don’t know how spiffsgen.py implements file system mirroring, and I don’t trust it. Here is the doc how to build the toolchain yourself:
Customized Setup of Toolchain

tius2000 · 2019-11-28T08:14:16Z

Hi @EaonL,

I had problems setting up a different toolchain on windows. So I tried to investigate the problem with spiffsgen.py and found a problem with the alignment of the page indices. spiffsgen.py seems not to do any alignment here. According to spiffs_nucleus.h SPIFFS_ALIGNED_OBJECT_INDEX_TABLES is not an integer value, but a binary flag:

#if SPIFFS_ALIGNED_OBJECT_INDEX_TABLES
    __attribute(( aligned(sizeof(spiffs_page_ix)) ))
#endif

As a test I added the following code to spiffsgen.py at line 220 and got a valid image :-)

#   Add padding if required
while len(img) % SPIFFS_PAGE_IX_LEN: img += b"\x00"

Of course, this should made a command line option for spiffsgen.py and enabled from Makefile.projbuild. Someone with more knowledge than I might provide a complete patch.

Thank you very much your help to find a solution!

GarMingLi · 2020-09-13T07:24:46Z

不知道为什么，我的环境中没有spiffsgen.py这个文件

EaonL closed this as completed Nov 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SPIFFS可以成功挂载文件系统，POSSIX / C均可以打开文件，但是读文件失败，返SPIFFS_ERR_INDEX_REF_LU [-10016] (GIT8266O-328) #766

SPIFFS可以成功挂载文件系统，POSSIX / C均可以打开文件，但是读文件失败，返SPIFFS_ERR_INDEX_REF_LU [-10016] (GIT8266O-328) #766

EaonL commented Nov 24, 2019 •

edited

EaonL commented Nov 24, 2019 •

edited

EaonL commented Nov 26, 2019 •

edited

tius2000 commented Nov 26, 2019

EaonL commented Nov 26, 2019 •

edited

tius2000 commented Nov 26, 2019

EaonL commented Nov 26, 2019 •

edited

tius2000 commented Nov 27, 2019 •

edited

EaonL commented Nov 28, 2019 •

edited

tius2000 commented Nov 28, 2019

GarMingLi commented Sep 13, 2020

SPIFFS可以成功挂载文件系统，POSSIX / C均可以打开文件，但是读文件失败，返SPIFFS_ERR_INDEX_REF_LU [-10016] (GIT8266O-328) #766

SPIFFS可以成功挂载文件系统，POSSIX / C均可以打开文件，但是读文件失败，返SPIFFS_ERR_INDEX_REF_LU [-10016] (GIT8266O-328) #766

Comments

EaonL commented Nov 24, 2019 • edited

EaonL commented Nov 24, 2019 • edited

EaonL commented Nov 26, 2019 • edited

tius2000 commented Nov 26, 2019

EaonL commented Nov 26, 2019 • edited

tius2000 commented Nov 26, 2019

EaonL commented Nov 26, 2019 • edited

tius2000 commented Nov 27, 2019 • edited

EaonL commented Nov 28, 2019 • edited

tius2000 commented Nov 28, 2019

GarMingLi commented Sep 13, 2020

EaonL commented Nov 24, 2019 •

edited

EaonL commented Nov 24, 2019 •

edited

EaonL commented Nov 26, 2019 •

edited

EaonL commented Nov 26, 2019 •

edited

EaonL commented Nov 26, 2019 •

edited

tius2000 commented Nov 27, 2019 •

edited

EaonL commented Nov 28, 2019 •

edited