Design notes for MoCaml test framework
- Dependency Injection
- Need a way to “install” the newly created test doubles
- This might be useful as a standalone library
- Might provide multiple solutions
- First class modules and references
- First class modules and runtime lookup
- Link-time implementation swapping
- Test Double Creation
- Automatically generate an implemetation of an interface
- Should handle:
- Function calls, both arguments and returns
- Counting events (at least once, exactly N times, etc.)
- How do we handle such rich expectations? We need more than just a hash table lookup, since we need to deal with number of times that an action is performed, etc.
- JMock will fail as soon as an expectation is violated, not after a validation step. I think we should do this too.
- Should handle:
[2012-10-05 Fri] Can we recover types in camlp4?
I don’t know that we can recover types from camlp4. By this I mean that if we parse a function that was written as ‘let f a = a+1’, since the parameter a does not have a type specified, we won’t be able to infer it from within a quotation. We could recover the types from module signatures, but that would require us to write signatures for all the modules we want to mock.
We could do two things. One, we could use ‘ocamlc -i’ for each module we want to mock so that we can get the signature of the module, and iterate over that. I’m not sure that this is the best solution, though, but it’s a start.
Two, we could perhaps just rewrite the mocked module with “empty” function bodies, and coerce the new module to be of the same signature as the original. The problem is that we won’t be able to build a datatype out of the direct types of the functions we want to mock. Perhaps we could just create polymorphic variants for each named function, and store values of these types in a table. I’m not sure how this would work, and it might make the syntax terribly cumbersome. We might be able to get around this, though. Something like:
module Foo = struct let func1 a b = a + b end module MockFoo = struct module type SIG_Foo = module type of Foo module Expect = struct type f_name = | F_func1 let exp = Hashtbl.create 10 let func1 f = Hashtbl.add exp F_func1 (`Ty_func1 f) end module Foo : SIG_Foo = struct let func1 a b = let `Ty_func1 f = Expect.(Hashtbl.find exp F_func1) in f a b end end
Wow, I just compiled the above code! This might actually work.
[2012-10-12 Fri] Progress
Found a good description of camlp4 tools by Nick Pouillard:
There are parsers, filters and printers. Parsers are for extending syntax, filters are for transforming ASTs, and printers are for printing ASTs. What we want to do is to take a file (probably an interface, but we might be limited to implementations) as input, inspect its AST, and output another file which implements the same interface as the first file. This sounds like we basically need a filter and a perhaps a printer.
It seems that filters can only transform signatures to signatures, or structures to structures, but not cross the two, which is what we want to be able to do: create a mock implementation of a given interface. The camlp4 binary has the option to treat any given input file as either an implementation or an interface, regardless of what the file actually is. This might be a way for us to get around the ‘a->’a nature of filters. Alternatively, there may be a way to use a filter to process an AST, and then use a printer to output our OCaml code. Perhaps the right way to do this is instead to use a parser as the frontend, but I’m not quite sure yet.
I do need to read the source code of type_conv, and read more code that uses type_conv. There may be something in it that we can use here.
Thought: can we pack an mli file into a dummy structure, and then filter over that using a str_item filter? Nope, not with ocamlc -pack, which requires a cmo file, and fails with just a cmi file.
Thanks to some mails from the OCaml list, I’ve got a way to filter over a signature, and then print out structure values. The key was to use Camlp4.PreCast.Printers.OCaml.print_implem to print the str_items we create, and use Obj.magic to cast the AstFilters.Ast types to PreCast types.
I’ve also fixed the bug in the structure filter implementation of mock_foo.ml I whipped up the other day. We can do a membership test on the Hashtbl, and if the function hasn’t been mocked, then we create another function which takes the right number of arguements, and then raises an exception of “No expectation found.” Though I just realised that the real test of this is when we don’t know the number of arguemnts in the function, so back to the drawing board on that – or really just give up, because mocking from an mli file will be much easier.
Idea for expectations
A mocked function should have the ability to record information in the expect module/object, such as the reason for a failure. So instead of just throwing an exception on error, we can do error reporting. Though we could just do failwith “foo”, or create a custom exception with the loggin data we need, so this might not be such a killer feature.
[2012-10-17 Wed] Success!
So I’ve worked out the filter to the point where we can generate an implementation file from an interface file. The whole thing isn’t done yet, but the structure of both the filter code and the output of the filter is basically finished.
We have a filter which iterates over a signature file, and builds up state in an object. The object has methods for generating the AST of the mocked module and the Expect module, which we use to set up the mock module for tests. Almost all of my initial mock_foo.ml skeleton has been implemented.
Here are some thoughts about this work:
- Using templates. I thought about using template ml files to hold code which could be included in the generated modules. Because these files couldn’t easily be compiled separately, since they were missing dependencies of their own, this didn’t work out. I also ran into problems with including a template module too early, and the Hashtbl type not being fully specified at that point. It just didn’t provide the benefit I was looking for.
- Inserting the mocked module’s name. I’m still not sure how we’ll do this, since the sig filter won’t know at filter time the name of the module type it’s operating on. My current idea is to use the ‘camlp4 -str <implem>’ option to pass in some fragment, say, ‘let module_name = FooModule’, on the command line, which will let us run a str_filter along with the sig_filter. We could pass in various options this way, I suppose. (Update: doesn’t look like this will work, because we can only operate on one sig/str object at a time. You can’t pass in a filename and a -str “”.)
- Objects. Turns out that objects are surprisingly lightweight to use. They seem to be the perfect thing to use here, since we need to build up state while we’re filtering, and then perform actions on that state while generating the new module.
- Using revised syntax. There are a few good references about revised syntax, but even they tell you to just print some sample original syntax out as revised syntax for some tricker bits. The command line tool to do this is ‘camlp4orf foo.ml -printer r’.
- Using revised syntax for function quotations. You really can’t do <:str_item< value f a = a >>. Just do <:str_item< value f = fun [ a -> a ] >>.
[2012-10-19 Fri] Version 0.1, and the way ahead
So I haven’t officially tagged this as version 0.1, but I think that this code is basically useable for generating mock modules from interfaces. It is by no means finished, though. While I do have a very simple test of the mock generator in this directory, it’s really only a toy example and not a great test case. I need to write more complicated tests, and also try filtering on a non-trivial mli file from xapi.
Some things I need to work on next:
- Inject the name of the module into the filter. As far as I can
tell, there is no way to recover the name of the mli file that is
being operated on. Some ideas I’ve had:
- Functorize the filter. We would pass in a module which would have the name of the module in some variable. We’d have to generate this module, and then compile the new filter, and then apply the filter to the mli file. We could generate this module as part of the ‘mock of’ syntax extention I’m planning to write.
- Post-process the generated ml file. We could just use ‘sed -i’ to replace a tag in the generated mli file with a real module name. I don’t really like this solution, but it’s easy.
- Find a way to recover the name of the mli file. I don’t know if this is possible, and I haven’t found a way to do it yet. Perhaps the list or #ocaml will know.
- Wait, perhaps we can get the file name out of the _loc? Yes, this is possible. Whew.
- Syntax extention ‘mock of module Foo’. We would use this extention in our unit tests. We would pass the resulting module in to the SUT as the new DOC. This would take care of generating the mock module file.
[2012-10-21 Sun] Testing on xapi_vm.mli
Some thoughts about shortcomings in the current implementation. We need to handle all sig_item cases appropriately, instead of just including the original module. This is 1) to avoid side effects that might be present in the original module (‘let _ = boom!’ at the top level), and 2) to be sure that we create mocks for everything in the interface file, not just the toplevel function definitions. We need to handle:
- nested modules (recursively filter)
- we might need multiple mock gen objects for this
- nested module types (easy, just copy)
- type and exception declarations (easy, just copy)
- external declarations (easy, just copy)
- module opens (easy, just copy)
- includes (could be tricky because we’d have to find that signature and process it too, inline)
- class definitions (disallow for now – no objects/classes)
- directives (ignore?)
Problems with compiling mock_xapi_vm.mli:
- No polymorphic variables in function defs. I’m not sure if there
is a way around this, but I think we may need to require that
interfaces define monomorphic functions. I hit this bug because
xapi_vm.mli (which was probably mostly autogenerated), erroneously
defines the function set_memory_dynamic_min to have a __context of
type ‘a (it should be type Context.t). When we try to create the
constructor T_set_memory_dynamic_min of (type of
set_memory_dynamic_min), we get an error on the type _context: ‘a,
because ‘a is an unbound type variable.
Is this something we could solve with GADTs? We could also take an entirely different approach to storing mock functions and use refs instead of Hashtables. This would mean that we would have a different ref for every mocked function, and we wouldn’t need the types f_name and f_type anymore, since the compiler would be able to infer the types of the references.
No, references don’t help because they suffer from the same restriction that Hashtbl does. I don’t know if it’s possilbe to use a GADT for this either. Perhaps encapsulating this in an object would be best. We might have to disallow abstract types in mocked functions, in which case we would have to just skip those in our definition, and include them from the original module. Let’s try testing mock_xapi_vm.ml now with any mocked polymorphic functions commented out.
- Woohoo, solution: We can enforce polymorphism using a record type with polymorphic…
[2012-11-07 Wed] Expectation feature idea
We should be able to set an expectation such that some function will be executed immediately after some mocked function has been called.
The need for this feature came up when I was debugging CA-48768, vm-import cancel. There was a timing issue where we were only checking for cancellation every ten seconds, so if you cancelled the operation ten seconds before it finished, the operation might continue on. The code in question is ocaml/xapi/import.ml:1329.
What I want is to be able to mock out Stream_vdi (among other dependencies), and after the call to Stream_vdi.recv_all, the expectation framework would cancel the task, and we would then make sure that the function ‘cleanup on_cleanup_stack’ was called.
This would be implemented in the mocked module. Each time we call a mocked function, we would store the result of the mock, and then execute all of the list of functions which are meant to be executed after the mocked function.
We don’t need the expectation framework to do this for us, though. We could just mock out the recv_all function, and have it cancel the task. This would, however, be a nice convenience to have, because it separates out the mock function’s functionality, and other side effects necessary for the test case.
(Another idea) For mocked functions which return unit, we should just have a default function which will return unit. This way we can automatically generate mock functions which only perform side effects.