-
Notifications
You must be signed in to change notification settings - Fork 37
"Deffert" - assert with promises in asynchronous tests #339
Comments
I like the idea, especially the sequencing is quite elegant. So, how would you test for an expected rejection? |
"some test" : function() {
return when(rejectedThingy()).
then(undefined, defert.equals("expected"));
} or "some test" : function() {
return when(asyncThingy()).
error(defert.equals("expected"));
} Buster will detect if no assertion is made. But one could be explicit about rejecting a resolve: "some test" : function() {
return when(asyncThingy()).
then(defert.never("asyncThingy must be rejected!"),
defert.equals("expected"));
} |
I see, thx. I'l have to think about it a bit, as I've been always thinking in terms of the alternative form for async tests, a Just one more question, maybe stupid: what is "defert" or "deffert"? My first guess is kinda portmanteau of "assert" and "deferred"..? |
Your first guess is correct :) It was sort of a working name that stuck with me and I forgot that it's not obvious. The name can be changed to whatever makes the most sense to the most people. What I'm mainly interested in is the mechanics of deferring an assert in this way. I would like something that plays a bit better with promises than the normal asserts. |
Come to think of it there is also a third path (onProgress). I think defert lends itself well for testing expected progress updates: return when(someThingWithProgress()).then(
defert.equals('final value'),
undefined,
defert.inSequence(
defert.equals('firstProgress'),
defert.equals('secondProgress'),
defert.equals('finalProgress'))); |
Glad to hear that, my next was something re "de-fertilizing"...
In an async test you wouldn't get a failure "No assertions", just a timeout. Correct me if I'm wrong, but that's one thing I found particularly annoying. So I guess what I wanna say is: not only could you be more explicit, you must.
Had been thinking about something like Here's a suggestion: let's first try to make precise what's to be addressed / what the problems are with what's currently available. At best without having any specific design idea in mind. That is, let's dive into "mechanics", or specs thereof. Once we have that we should be able to judge/compare specific ideas better.
Right, hadn't been thinking of this. It's rather tricky/specific though, as e.g. how many progress updates would you require exactly? What really (practically) would be the thing to (reasonably) assert on in it? progress pct being between 0 and 100? What's not so elegant IMO is adding Note that these are all just some first thoughts of mine, TBD. |
About detecting assertions. When I return a promise I never get a timeout (unless the promise is never either resolved or rejected) but I do sometimes get a weird behavior. It may be related to no assertions. buster.testCase("promises", {
"rejected test" : function() {
var defer = when.defer();
window.setTimeout(function(){ defer.reject("hello")}, 100);
return defer.promise;
},
"resolved test" : function() {
var defer = when.defer();
window.setTimeout(function() {defer.resolve("hello")}, 100);
return defer.promise;
},
"other" : function() {
return when("hello");
}
}); This results in: [johlrogge@kirishima phloem]$ buster test promises -r specification As you can see, no timeout and WARNING: No assertions. The undefined:undefined is new to me. I know that I have seen this work at some point so it may be a new bug (I recently updated buster). I do think that the timeout you experienced has to do with the "done" way of doing things. I'll address your other points after work. This one took longer than I thought :) |
These tests behave correctly "resolved assert" : function() {
return when("hello").then(function(val){buster.assert.equals(val, "hello");})
},
"resolved assert" : function() {
return when("hello").then(function(val){})
},
"resolved assert" : function() {
return when("hello").then(function(val){throw new Error("hello")})
} |
First of all: I'm loving this idea! Looks very nice. Not crazy about the name "defert" though - I like the idea of it, but it looks too much like a spelling mistake to me :) I'm thinking this could be built as a standalone module, an extension to referee. Generating a deferred wrapper can probably be done in a unified way (e.g. a loop and one function) once we settle on the API. Can we find some way of expressing this within the existing API? E.g. |
Thanks! Yes, defert is not the essence of the idea. I think assert.deferred or deferred.assert works well. Perhaps deferred.* would be a good place to collect async utils? Expect to be is sort of wired in the wrong way for splitting the value up since the actual goes first and the API makes sure you cannot get it "wrong". Also future tense does not fit in the promise vocabulary. The "defer.assert.*" keeps some of the benefits of expect (the dictated order of actual and expected) but loses some of the BDD "words matter"-beef. Perhaps something like this is in the right direction: when(24).then(expect.actual.toBe(24)); Of course not all of them, just not locking onto the first word I thought of. I do think the 4th alternative is not good enough. I would like a noun in between... |
About something for both camps: The benefit is more obvious in the promise camp. As it is now the current tools do leave promises at a disadvantage. However, what you do get in either camp (and that maybe makes "deferred" a bad choise of concept to package the idae in) is: composability You can build higher level asserts with this technique even if you don't use promises: var assertSuccessful = defer.assert.match({status:"successful"}); that can be used like this: assertSuccessful(service.getResult()); Also, as seen in the progress these kind of asserts can be composed. Here is another example: var definedSuccess = defer.assert.and(
defer.assert.defined,
defer.assert.equals("success")
) The example is a bit contreived but you get the idea. This kind of composability is most useful again when you test asynchronous code with promises. It is however not useless in "normal" tests. @cjohansen I could start a project for an extension if you don't have something else in mind (feel free to use the idea in referee and integrate it in buster that way). |
@johlrogge: Sounds like a good way to start. Btw, I would love for this module to be part of the default Buster stack. My suggestion to develop it as a separate module is just because I think that makes sense organizational-wise. When you have some code we can decide whether it's most beneficial to keep it a separate module or just include it in referee. I totally agree on promises being at an unfair disadvantage. Solving this will be a great win. |
Any good ideas for a module name. I know what we don't want :) |
Something simple and straight-forward? |
BDD: Module name: Composability is a big asset, that's for sure. Just so I can get my head wrapped around it: the basic idea is partially applying the assertion, but not from left to right as usual. Rather it's the first argument ("actual") which is "postponed" since that is what we get last. The main problem with async tests is IMHO this: the nice linear (and flat) sequence of arrange-act-assert tends to be lost. Mostly it's the "act" and "assert" that get interleaved, and often things become nested at several levels. |
@meisl A partial application that starts from the second argument is sort of what's going on I think. Also, this is mainly intended to help with promised-based APIs, for which it is highly useful. I think we have to target various specific usecases like that to eventually be allround good for async tests. BDD: |
@meisl Sorry for the late reply. I was really busy yesterday. The main goal of this is to "get inside" the promise resolution and assert there. Partial application of what we know before executing the test (such as which values we expect the test to yeild) is what I'm suggesting. (You need to get a one argument function into the return sequential(promise,
deferred.assert.defined,
deferred.assert.matches({key:'value',
id: deferred.assert,greaterThan(0)})); Notice the nested assert inside the matched structure also notice that such an assert can be saved for reuse: var assertValidObject = deferred.assert.matches({id: deferred.assert,greaterThan(0)}); You can also to nifty things with return when(promise).
then(_.tap(_.pluck('key', deferred.assert.greaterThan(0))).
then(...) Partially applied asserts opens up for very innovative (and easy to understand) testing styles that IMHO are way underused in testing frameworks. Most frameworks start off in the xUnit imperative school and tends to get stuck there. The imperative style tends to get really messy for asynchronous stuff but actually also hard to reuse parts even in synchronous tests.
To have expect automatically resolve promises is also a good suggestion. Actually making referee promise-aware in this way is another good idea. When in cujojs has a lot of overloading going on so that you can either do when(12) or when(promise). Making assert, refute and expect work this way would be a cool thing. I'm not quite clear about how the assert would signal that the test is finished though (as is usually done with My $500 ;) |
I noticed we started to get to the core of the usefulness of the suggested feature (Thanks @meisl for poking them out with questions). Partially applied asserts are not only useful for asynchronous testing which is what I associate defer with (even if what we technically do is deferring the "actual"). I'm tossing referee-partial or partial-referee into the pile of possible names. More implementation specific, less application associated (?) |
Interesting. I like both of those. Maybe naming magician @magnars has a suggestion that captures this essence in some creative way? :) |
Hey, this looks pretty nice. I like the If you want some clever name, how about |
@magnars |
@cjohansen Then I'd say that |
Ok, /J |
:D that's a good one! Although I'm not so sure about it in view of the recent scandal in football bets... @johlrogge: thanks for the explanations and examples. Glad I'm not getting on your nerves too much :)
I see. But still, can't we have it construct a thing with asserts inside, only under the hood? expect(promise).eventually.toBeDefined()
.and.itsProperty("key").toBe("aKey")
.and.itsProperty("id").toBeGreaterThan(42); The point being: instead of sequencing several |
Maybe I should have said that of course I expect @cjohansen: could you give me some recommendation where to look for how the BDD stuff is working (internally)? @johlrogge: I'm not sure I understand what you meant by orthogonal (~> signal that test is finished). Does it mean that your proposal, as it stands, and without my speculations, could be used with buster as is? That is, would not require to make buster track started assertions and check if all eventually succeed? |
@meisl about orthogonal, it got pushed out of context. What I meant is that being clever about promises is orthogonal to inserting inside promise chains. I can see a value in both. |
Yep, thx. No need to haste :) @cjohansen: nevermind, stupid q. |
Remark on |
@johlrogge I'm thinking depend on referee, then export a module that extends it? So, to use these assertions, one would |
opinion That is exactly why I didn't want async. The purpose is to be able to build partially applied assertions. Partially applied assertions, I have come to realize, very much applicable outside of the asynchronous domain. A good name, IMO, should not limit the end users imagination. That is the problem with abstract concepts, they are applicable over a plethora of domains and hence should be called exactly what they are: the name of the abstract concept.
Never ;)
Yes (sorry)
Just to be clear, this has nothing to do with implicit (at least from what I know from scala and what the word in general means to me). Left out is more correct IMO.
It looks like that but remember that this kind of boiler plate can be reduced one char and a dot It is also important to note the number of things to keep in mind without partial asserts:
But most important is that partially applied asserts is a different kind of animal with more levels of flexibility. If this was scala and not javascript it wouldn't be needed, I could just do this: assert.equals(_, expected) and I would be good to go. I if expected was the first argument (which it is in junit for instance) I could get a way with _.partial(assert.equals, expected) which is a bit better. Actually none of the two would solve chaining unless we changed assert.* to return their actuals...
Yes the above is the object oriented builder way of doing things. Nothing wrong with that. It generally requires some meta programing voodo to make alla options availible for proper chaining. I'm not a fan of this style since it is pretty constrained to what the framework author had in mind and requires reading a manual to extend it. That is just my opinion though. I think the functional approach composable in a natural way. You mentioned specs, did you mean the scala testing framework or specs as opposed to tests? I think rspec is probably a more suitable source of inspiration for various reasons, one of them being that it is written in a dynamic language with more similar constraints/posibilities. conclusion I'm leaning towards I can whip something together pretty soon I think but my evenings are a little crammed so not sure about this week. I think I have enough to get started anyway :) |
@cjohansen Ok, I'll do that :) |
Ok, quite some things. For now I'll go for those that only need clarification.
Didn't have anything Scala in mind. Just as opposed to "explicit", ie. no symbol for it, but still being a crucial part, through context. No problem with calling this "left out".
Specs as opposed to tests, no Scala.
Very good points, thx for pointing them out. Didn't see it.
Well, higher order functions and composability are known to go together pretty well. Scala's conditions of coming into existence are undoubtedly a lot more fortunate than those of JS. Nevertheless, your comparing the two is valuable. THE RANT:
Good point, especially the last sentence. So if the module in fact only comprises the partial app stuff - which makes a lot of sense - then "async" is inappropriate. It has to be differentiated from other options to deal with async tests.
Well, it must to some extent, if it's to convey anything. Take for example an extreme: Now let's put There's a tendency, especially in the JS community, that I, personally, find pretty hard to cope with. It's simply that it seems everyone's trying to be particularly "clever" with names for their stuff. With "clever", you'll guess it, being not exactly defined as I would define it... |
@meisl I applaud you, good sir. That rant was spot on. Thanks! |
combinatorsExcellent observation! I really see a resemblance between parser combinators and "assertion combinators" to the point that I am very excited by the combinators angle of it and am starting to think that "combinator" actually captures the essence of the idea. It just happens to also be very applicable for promises. namingI should have said "should not unnecessarily limit the users imagination". Because of the path into this thread (pain when testing with promises) I initially took the spin of "defer" which I feel overly constrained my own imagination about the concepts applicability. That was my point. The first rule of names: Until you know, pick a bad one or it will stick :) Defert was maybe too good to start with but it was never something I argued strongly for :) About cleverness: it was not until impartial-referee was suggested that I actually thought of the duality of partial-referee. I didn't intend to be clever I wanted to convey: relationship with referee hence referee, and partial as in *partial application hence partial. The cleverness made me lean towards partial-referee over referee-partial since it is more memorable Lets not forget: names should be descriptive but they are not descriptions. There is no way we can pick a name that conveys the whole purpose of the framework but a name certainly does lock some neurons which is why I wanted to be more open than just defer. The tendency in the JS community to be funny is not a problem for external names and it is not only in the JS community. Take Java for instance: Spring, Guava, Guice, Hamcrest to name a few. Are they bad names? No, they are memorable and googleable just like busterjs and there is no way that a single word or even two can be picked to properly describe guava for instance. And to be fair: clever names are more memorable once you get the cleverness. In this case however we are talking about an internal name. An extension to referee so we can piggyback on referee. So relationship should therefore perhaps be primary referee-x - fair enough. That would give referee-partial. Now I have to suggest refreree-combinators, my current favorite. My opinions of names are strongly phrased weakly held :) (But I /DO/ care about the naming or I would not spend this much time debating it) |
+1 Re: Naming strategies. I agree with what @meisl writes to a large extent when it comes to naming concepts in code. However, as @johlrogge says - module names are a little bit different. It is not a full description. Module names also take on the role of the "identity" or "personality" of the module. Generally I try to pick names that have a "namey" sound to them ("Buster", "Sinon", "Referee" etc) and that has some back-story/semantics that is meaningful within the context of what the module does. The hope is that these two aspects help me find memorable names.
I agree a million times with this. |
Thanks to all, what a great discussion!
Btw: here's where the (my) problems with Re combinators: in fact I was thinking only of combinatory logic (CL) but parser combinators - or maybe even general monads? - might provide even more/better inspiration.
Excellently phrased, in my opinion! [1]: with 2 "r"s in |
Lets set the bar high! And if anyone whines we make them help us :)
That is very interesting. I'll try to set up a project now and get Starting now, hope to have something this weekend. |
Had an issue depening on referee:
The checked in version depends on lodash but when installing from npm it tries to install underscore. Perhaps release 0.11.1 ? |
I have a skeleton up and running. The basic case works in node.js I had to clone referee manually and npm link it before I could get my package.json to work. See my error in the previous comment (I think referee is broken in NPM) /cc @cjohansen Anyway, The case that started this discussion is implemented. I have not tested all asserts but I tested one with expected actual and one with just actual. I think I should also override add so that extending referee with assertions will automatically add combinator versions. I have grand plans for making combinators that makes it easy to dig into structures and poke out some data you want to verify. I think it will become kind of a Like: assert.structure({id: assert.greaterThan(12),
posts: assert.some(assert.structure({read:true}))}); Asserting "everything at once" in a structure assert is probably overkill and bat practice. The benefit is that you can poke out a small amount of data from a structure (pretty much like with the power of match) but be more specific about how to handle nested structures with arrays etc. The structure assert should pretty much just be sugar that generates several attribute-asserts. I think dusting off my parser combinators will help a great deal to find the right basic elements. One can even imagine that with combinable asserts one can build schemas for validating forms etc which could extend the usefulness of referee... That is however secondary. Have a look :) |
Fantastic :D will check out ASAP! Re npm I'm not the right person, but I think it's fine to go with the git HEAD of referee for now. It's not unlikely that some adjustments in referee will be needed. Also, not even the latest buster on npm is depending on it (still buster-assertions there). Re "steroids": what's that? I'm thinking "Lance Armstrong" / anabolica but I guess that's not quite what you mean...? Re CL: it's the fraternal twin of lambda calculus, so it's the foundation of everything, so to speak. It's also very basic[1]... "combinable asserts" - that's exactly the point! In my speak it's "there must be something like an Assertion monad". Since you mentioned Scala, where do you know "parser combinators" from? Oh, this is really getting exciting :D I really have to finish now the stuff I was working on in ramp-resources... [1]: I think I will turn my scribblings into a little "CL 101" and post it here. One can always ignore it if it's nothing new or boring. |
For everyone who's similarly stupid as I am - it's here: https://github.com/busterjs/referee-combinators |
About steroids I mean When I put some effort into learning parser combinators I was using Scala and I found this blog particularly useful |
Re steroids: sorry, looks like there's something wrong with me. Had to watch and wonder what they are on in order to get it. But it does show that my first guessings aren't always that bad, after all... Re the blog on parser combinators in Scala: thanks a lot for that! Looks very interesting and well-written, albeit quite voluminous. So... |
"Theory - that's what you don't understand. Practice is what you cannot explain." |
@johlrogge Sorry about the referee screwup. Publishing 0.11.1 as we speak. Also, exciting work. Really looking forward to that assert.structure thing :) |
Congratz @johlrogge, you've got your first pull request ;) I also added a README where I wrote down how I worked around the referee issue - should be obsolete by now. |
Hi guys. I've pushed the tests a bit further in the direction started by @johlrogge, namely (more) similar to how assertions are tested in referee itself. I turned around a few things so the appearance is now a bit different. I made preliminary pull request #2 out of what I did so far. But beware, it's far from finished! Rather so we have a place to discuss details / how to proceed. Plz have a look. |
|
...not really understanding anymore what this was about - "finally" is probably what I want to say :) |
Buster has a really nice feature: return a promise for an asynchronous test:
I came up with a pattern to make asserting simpler (I know at least one other person has done something similar so probably pretty obvious):
Basically it allows splitting asserts in two parts: first declare your expectations. Each expectation returns a function that accepts the actual value and performs the assert.
Each assert also returns the actual value so that several assertions can be made in sequence:
I don't know if there is any other way I missed to do this with existing assertions or if this would be a good extension (or part of the buster core assertions)?
The text was updated successfully, but these errors were encountered: