Directory lookup for test-runner on OS X

[paulfryzel]

The test-runner is failing when looking up test directories. This appears to be another OS-related problem; this time with readdir/dirent.

$ make test
...
gc/stackvm
gc/load
FFI
Starting test-runner.js...
 ---
run-time error: run-time error: Unable to open directory: ./tests/..
$rt_throwExc(2FA842) ("runtime/runtime.js"@128:1)
dir_prototype_open(32C434) ("lib/dir.js"@95:32)
dir(32C36E) ("lib/dir.js"@87:23)
anon(32C4F9) ("lib/dir.js"@173:39)
runTests(30A5E7) ("test-runner.js"@117:19)
anon(30A6F5) ("test-runner.js"@152:13)
Function_prototype_apply(2FF388) ("stdlib/function.js"@226:18)
Function_prototype_call(2FF3A1) ("stdlib/function.js"@245:18)
array_forEach(3011BD) ("stdlib/array.js"@578:9)
anon(30A528) ("test-runner.js"@149:5)
test_runner_js(30A522) ("test-runner.js"@43:2)
       52.90 real        51.93 user         0.61 sys
make: *** [test] Error 255

On Darwin, the struct's used to store the directory entries differ from their Linux counterparts:

struct dirent { /* when _DARWIN_FEATURE_64_BIT_INODE is NOT defined */
    ino_t      d_ino;                /* file number of entry */
    __uint16_t d_reclen;             /* length of this record */
    __uint8_t  d_type;               /* file type, see below */
    __uint8_t  d_namlen;             /* length of string in d_name */
    char    d_name[255 + 1];   /* name must be no longer than this */
};

struct dirent { /* when _DARWIN_FEATURE_64_BIT_INODE is defined */
    ino_t      d_fileno;     /* file number of entry */
    __uint16_t d_seekoff;    /* seek offset (optional, used by servers) */
    __uint16_t d_reclen;     /* length of this record */
    __uint16_t d_namlen;     /* length of string in d_name */
    __uint8_t  d_type;       /* file type, see below */
    char    d_name[1024];    /* name must be no longer than this */
};

[mollymorphic]

Cool, for the time being we can fix this by using ffi.os and creating two different cdef blocks for the dirent struct (and any other declarations that need to be different on Linux vs Mac). Thanks for figuring this out.

[paulfryzel]

Hmm, @teebrz there seems to be some discrepancies between what the header docs are asking for and the data I'm seeing come back.

First, I've lightly instrumented dir#getDirs():

dir.prototype.getDirs = function()
{
    var ent;
    var dname;
    var dirs = [];
    while (!ffi.isNullPtr(ent = c.readdir(this.handle)))
    {
        ent = c.dirent(ent);
        dname = ent.d_name.toString();
+       print(">>>", dname, "<<<");
        if (dname === '.' || dname === '..')
            continue;
        if (isDir(ent))
            dirs.push(dname);
    }
    c.rewinddir(this.handle);
    return dirs;
};

... and created relevant typedef's:

typedef unsigned long long  __uint64_t;
typedef unsigned int        __uint32_t;
typedef unsigned short      __uint16_t;
typedef unsigned char       __uint8_t;

With the dirent struct for my OS/arch DARWIN_FEATURE_64_BIT_INODE=1 , where __ino_t maps to __uint64_t:

struct dirent

__uint64_t d_fileno;
__uint16_t d_seekoff;
__uint16_t d_reclen;
__uint16_t d_namlen;
__uint8_t  d_type;
char       d_name[1024];

output

Starting test-runner.js...
 ---
>>>.<<<
>>>0-runtime<<<
>>>Í<<<
>>>Í<<<
>>>Í<<<
>>><<<
>>><<<
>>>Í<<<
>>><<<
>>>Í<<<
>>><<<
>>>Í<<<
>>><<<
test-runner.js results:
 ---
Tests run:      0
Tests failed:   0

Removing the optional d_seekoff:

struct dirent

__uint64_t d_fileno;
__uint16_t d_reclen;
__uint16_t d_namlen;
__uint8_t  d_type;
char       d_name[1024];

output

Starting test-runner.js...
 ---
>>><<<
>>><<<
>>>e<<<
>>><<<
>>>s<<<
>>>ests<<<
>>>ench<<<
>>>ut<<<
>>>der<<<
>>>h<<<
>>>Í<<<
>>>h<<<
>>><<<
test-runner.js results:
 ---
Tests run:      0
Tests failed:   0

Giving things a try with the 32-bit setup, where __ino_t maps to __uint32_t:

struct dirent

__uint32_t d_ino;
__uint16_t d_reclen;
__uint8_t  d_namlen;
__uint8_t  d_type;
char       d_name[256];

Starting test-runner.js...
 ---
>>><<<
>>><<<
>>>-runtime<<<
>>>-stdlib<<<
>>>-regress<<<
>>>-misc-tests<<<
>>>-misc-bench<<<
>>>-shootout<<<
>>>-sunspider<<<
>>>-v8bench<<<
>>>-lib<<<
>>>-demos<<<
>>>re<<<
test-runner.js results:
 ---
Tests run:      0
Tests failed:   0

The closest I was able to get was with a seemingly random definition:

struct dirent

__uint32_t d_ino;
__uint8_t  d_reclen; // __uint16_t => __uint8_t
__uint8_t  d_namlen;
__uint8_t  d_type;
char       d_name[256];

output

Starting test-runner.js...
 ---
>>>.<<<
>>>..<<<
>>>
00-runtime<<<
>>>     01-stdlib<<<
>>>
02-regress<<<
03-misc-tests<<<
04-misc-bench<<<
>>>
   05-shootout<<<
>>>
   06-sunspider<<<
>>>
07-v8bench<<<
>>>08-lib<<<
>>09-demos<<<
>>>core<<<
run-time error: run-time error: Unable to open directory: ./tests/.

Finally... complete directory output, but padded with various ASCII chars in the 0-10'ish range. Any ideas on what could be going on here?

[mollymorphic]

Hm, not sure except it's some kind of issue with struct layout/alignment - the ffi is trying to read the struct incorrectly or the struct it gets back doesn't look like it expects it to or something like that. I'll take a look and see if I can find anything, thanks for looking into this.