tests framework

[HarlanH]

As hashed out in #8, I've put together a stab at a test framework that can be used for unit testing as well as (eventually) build testing. Note that right now this test framework is not used at all when you do make test.

To see how it works, compile, then do julia -e "tests()", which will run any files called test_*.jl in or under the current directory. Note that the printer prints a dot for successful tests and dumps a structure for failures. See test\test_test.jl to see the example test suite.

Issues (tagged with TODO):

1. The default printer, test_printer_simple, needs to be slightly spiffed up to produce user-friendly text output.
2. I had issues when defining a local struct inside of a macro and then accessing its fields. Probably a syntax error, but I couldn't make it work, so I just define a bunch of local variables instead. Works fine, looks ugly. Someone should fix it (and tell me what they did!).
3. I get an inexplicable error indexing an args array. I commented out that line. Someone smarter than I should take a look to see why the 4th argument in a 4-argument expression can't be accessed.

Aside from that, it was a pleasure to write this in Julia. Great language!

[HarlanH]

Oh, and as mentioned in #560, file and directory tests aren't exposed yet. I used a silly solution involving the stat command, but eventually those should be replaced with proper wrappers around system calls.

[nolta]

Fails on linux:

$ julia -e "tests()"
stat: cannot read file system information for `%HT': No such file or directory
failed process: `stat -f %HT .`
 in _readall at /scratch/nolta/julia/jl/process.jl:477
 in readall at /scratch/nolta/julia/jl/process.jl:483
 in tests at /scratch/nolta/julia/jl/test.jl:6
 in tests at /scratch/nolta/julia/jl/test.jl:23
 in process_options at /scratch/nolta/julia/jl/client.jl:136
 in _start at /scratch/nolta/julia/jl/client.jl:207

$ julia test/test_test.jl 
pop: array is empty
 in pop at /scratch/nolta/julia/jl/array.jl:449
 in produce at /scratch/nolta/julia/jl/task.jl:23
 in include at src/boot.jl:192
at test/test_test.jl:8
 in include at src/boot.jl:192

[HarlanH]

Ah. The former error must be because the stat command is different on Linux. Weird and annoying. Well, we have to have the ability to test whether a file is a file or a directory, which is #560. Maybe someone should work on that?

The latter is not currently unexpected. The test scripts are supposed to be loaded in a Task, and if they're not, I'm not surprised that they fail. Ideally I guess they shouldn't. But for now, to test a single file, tests("test/test_test.jl") should work. (At least on MacOS X).

[pao]

You might just wait on (or build on top of) #521. There's an abstraction for stat in libuv.

[JeffBezanson]

Thanks for the code, and the compliment!

The syntax $a.b is parsed as $(a.b), which is probably the source of the macro problem. With $ you often need lots of parens, in this case ($a).b. Quote/unquote is annoying with fancy syntax. So typically with macros we try to do as much as possible outside quote; basically using the macro to pluck out interesting bits of the syntax, convert things to strings and/or closures, and pass it all off to a function as soon as possible, so most of the actual code can be written as normal code not in quote.

The args array indexing thing sounds like something that should be filed as a separate issue.

[JeffBezanson]

Also, I may not have read it thoroughly enough to fully understand what's going on, but you generally don't want to call eval in a macro. In a macro you place an expression in the output code, and it will be evaluated just by virtue of being present in the code as it runs.

Also it is generally preferred style in macros to evaluate things as soon as possible. For example whether takes_less_than is present is entirely static, so that comparison can be done outside the quote.

[Keno]

For linux you would want to do

stat -c %F

Also, you'd have to take the lower case of everything to make it work on all system as the Linux output is lower case.

Regarding libuv, it has a stat abstraction, but I have not implemented the FS abstraction within Julia yet (mainly because Julia's FS IO just works\tm on Windows), but I'll do that as soon as the rest is done.

[HarlanH]

OK, thanks all.

It sounds like I should wait on the file system stuff until libuv settles. In the meantime, I'll comment out tests() except for the version that takes a single filename argument, so that we can test the rest.

Jeff, I'll see if I can figure out how to make the macro more Juliaesque (Juliaish? Julie?). I tried that initially, but had trouble figuring out how to pass an unevaluated expression to a function from a macro. The :($x) syntax didn't seem to work, but maybe I did something else wrong.

And I'll put together a simple demo of the args array indexing issue and file it as a bug.

[HarlanH]

tests("test/test_test.jl") now gives the expected successes and failures for me. The code is considerably cleaner than before -- thanks!

[ViralBShah]

@StefanKarpinski Can you review? I think a test framework will take a few iterations to get right, but we need to get started.

[StefanKarpinski]

Yeah, sorry, I've been meaning to go over this and keep getting sidetracked...