Unreliable pnor access through /dev/mtd

[cyrilbur-ibm]

I've been doing some work which relies on /dev/mtd7 "pnor" and using it has been unreliable. I've essentially been wanting to put all of the pnor into ram or into another file.

Depending on the kernel version different things happen, with:

 uname -a  Linux palmetto 4.7.3-be2001133835be80ba4657232b823ba2daaac0d6 #1 Wed Sep 14 12:05:02 UTC 2016 armv5tejl GNU/Linux

My test code succeeds, once. The second time it reads all 0xff into the file.

With a recent rework by Cedric there's a nice oops:

./mtd_test /dev/mtd7
Using /dev/mtd7
mmaped /tmp/mtd_test.out to 0xb2f82000
Reading from /dev/mtd7 to /tmp/mtd_test.out (0xb2f82000) for 0x04000000
Unable to handle kernel paging request at virtual address e6000000
pgd = dd7f8000
[e6000000] *pgd=00000000
Internal error: Oops: 5 [#1] ARM
CPU: 0 PID: 817 Comm: mtd_test Not tainted 4.7.10cyrilbf528b5ac8dc9148be599117b1db769079f58face #1
Hardware name: ASpeed SoC
task: dd7dac20 ti: dd7ce000 task.ti: dd7ce000
PC is at aspeed_smc_from_ahb+0x50/0x94
LR is at 0xddc00000
pc : [<c02d2fc0>]    lr : [<ddc00000>]    psr: 20000013
sp : dd7cfdf4  ip : e6000000  fp : 00000000
r10: 02000000  r9 : ddc00000  r8 : dd7cfe94
r7 : 00400000  r6 : 02000000  r5 : de512010  r4 : 00400000
r3 : 00400000  r2 : 00400000  r1 : e6000000  r0 : ddc00000
Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
Control: 0005317f  Table: 5d7f8000  DAC: 00000051
Process mtd_test (pid: 817, stack limit = 0xdd7ce190)
Stack: (0xdd7cfdf4 to 0xdd7d0000)
fde0:                                              c02d35cc de512038 c02d3524
fe00: de512038 00000000 02000000 00400000 00000000 c02d198c 00400000 dd7cfe94
fe20: ddc00000 00000000 de512038 dd7cff78 02000000 00000000 dd80be20 c02c9ee8
fe40: 00400000 dd7cfe94 ddc00000 00000000 02000000 de512038 b4f82000 dd80be20
fe60: ddc00000 02000000 00000000 c02cf4ac 00400000 dd7cfe94 ddc00000 00000100
fe80: 60000013 02000000 dd7ce000 c060200c 00000048 00000000 00400000 c012da74
fea0: 00000004 00000001 de793c00 c029b0e0 dd4e63c0 de6fdd80 c029f580 bd9d30d1
fec0: de76cdc0 c02cf2c4 04000000 c060200c b2f82000 dd7cff78 04000000 00000000
fee0: befe4cb4 c01a15bc dd767d10 dd767d24 dd767d20 bd9d30d1 00000002 000000ff
ff00: 04000000 de76cdc0 dd469dc8 00000001 00000007 00000000 dd4e63c0 00000000
ff20: 00000000 bd9d30d1 00000000 dd469b40 dd7cff78 bd9d30d1 dd469b40 04000000
ff40: 00000000 b2f82000 dd7cff78 c01a174c dd469b40 b2f82000 04000000 dd469b40
ff60: 00000000 c060200c dd469b40 b2f82000 04000000 c01a1b5c 02000000 00000000
ff80: 00000000 bd9d30d1 00000001 00010908 00000000 000104e4 00000003 c0102344
ffa0: dd7ce000 c01021a0 00010908 00000000 00000003 b2f82000 04000000 04000000
ffc0: 00010908 00000000 000104e4 00000003 00000000 00000000 47f00000 befe4cb4
ffe0: 00000000 befe4c6c 000107e8 47fd308c 60000010 00000003 5effd871 5effdc71
[<c02d2fc0>] (aspeed_smc_from_ahb) from [<c02d35cc>] (aspeed_smc_read+0xa8/0xd8)
[<c02d35cc>] (aspeed_smc_read) from [<c02d198c>] (spi_nor_read+0x74/0x98)
[<c02d198c>] (spi_nor_read) from [<c02c9ee8>] (mtd_read+0x64/0x9c)
[<c02c9ee8>] (mtd_read) from [<c02cf4ac>] (mtdchar_read+0x1e8/0x240)
[<c02cf4ac>] (mtdchar_read) from [<c01a15bc>] (__vfs_read+0x28/0x128)
[<c01a15bc>] (__vfs_read) from [<c01a174c>] (vfs_read+0x90/0xfc)
[<c01a174c>] (vfs_read) from [<c01a1b5c>] (SyS_read+0x4c/0x9c)
[<c01a1b5c>] (SyS_read) from [<c01021a0>] (ret_fast_syscall+0x0/0x38)
Code: e1510002 1a000005 e3a00000 e49df004 (e59cc000) 
---[ end trace fcfa3868667a9aed ]---
Segmentation fault

Note: I believe there was a v2 of Cedrics patches that were actually applied, I will retest with these.

Test code:

#define _GNU_SOURCE
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <mtd/mtd-abi.h>

int main(int argc, char *argv[])
{
	struct mtd_info_user mtd_info;
	int mtd_fd, tmp_fd, rc;
	char *tmp_ptr;

	if (argc != 2) {
		fprintf(stderr, "Usage: %s /dev/mtdx\n", argv[0]);
		return 1;
	}

	printf("Using %s\n", argv[1]);
	mtd_fd = open(argv[1], O_RDONLY, 0);
	if (mtd_fd == -1) {
		perror("Opening mtd");
		return 1;
	}

	rc = ioctl(mtd_fd, MEMGETINFO, &mtd_info);
	if (rc) {
		perror("MEMGETINFO ioctl");
		close(mtd_fd);
		return 1;
	}

	tmp_fd = open("/tmp/mtd_test.out", O_RDWR | O_CREAT | O_TRUNC, S_IRWXU | S_IRWXG | S_IRWXO);
	if (tmp_fd == -1) {
		perror("Opening tmp file");
		close(mtd_fd);
		return 1;
	}

	rc = fallocate(tmp_fd, 0, 0, mtd_info.size);
	if (rc == -1) {
		perror("fallocate");
		close(tmp_fd);
		close(mtd_fd);
		return 1;
	}

	tmp_ptr = mmap(NULL, mtd_info.size, PROT_READ | PROT_WRITE, MAP_SHARED, tmp_fd, 0);
	if (tmp_ptr == MAP_FAILED) {
		perror("mmaping tmp file");
		close(tmp_fd);
		close(mtd_fd);
		return 1;
	}
	printf("mmaped /tmp/mtd_test.out to %p\n", tmp_ptr);

	printf("Reading from %s to /tmp/mtd_test.out (%p) for 0x%08x\n", argv[1], tmp_ptr, mtd_info.size);
	rc = read(mtd_fd, tmp_ptr, mtd_info.size);
	if (rc == -1) {
		perror("read");
		munmap(tmp_ptr, mtd_info.size);
		close(tmp_fd);
		close(mtd_fd);
		return 1;
	}
	if (rc != mtd_info.size)
		fprintf(stderr, "Short read %d out of %d\n", rc, mtd_info.size);

	if (*tmp_ptr == 0xff)
		fprintf(stderr, "The first byte of the flash read was 0xff, this can't be good\n");

	printf("You can now inspect what was read from %s in /tmp/mtd_test.out\n", argv[1]);
	munmap(tmp_ptr, mtd_info.size);
	close(tmp_fd);
	close(mtd_fd);

	return 0;
}

[shenki]

I tested on a Palmetto with a platform a mx66l51235l (65536 Kbytes) and could not reproduce. The flash contents looked good to me.

4.7.10-0edfa645374197350f6e64e32ef142bb4ab41b31

[cyrilbur-ibm]

Appologies for the delay!

I couldn't find that sha in my tree.

I've done testing against: 9e654d6

Reliability of /dev/mtd7 has greatly increased, I've still had it return all 0xff in my buffer while succeeding the read() call. This was detected by the mtd test program above. Unfortunately it is quite rare now and I don't seem to find a good way to reproduce.

I have tried:

• Concurrent accesses to /dev/mtd7 (two mtd tests running - modified to not use the same output file)
• Stressing the CPU from userspace while running mtd test
• Booting the host (with my box changes)
• reboot
• reboot -f
• running pflash (mtd access only) intermittently

Perhaps my flash chip is unreliable?

[legoater]

could you a run a "md5sum /dev/mtd7" in a loop ?

[legoater]

I have run some mtd-utils torture tools on the evb and the palmetto and could not reproduce.
Here are the binaries for OpenBMC if you want to give a try : http://kaod.org/openbmc/mtd-utils/

Unfortunately, I can not boot the palmetto for the moment. But that is a good test.

[legoater]

Is this still an issue ?

[shenki]

Lets close