testing: prove is obsolete and archived

[digikar99]

The author has archived Prove and The Common Lisp Wiki mentions it to be obsolete.

Guess we need a newer recipe. But more importantly I think, prove needs to be taken down from the cookbook. Either that, or seeing the number of stars on prove and fiveam, prove needs a new maintainer. Are there any download statistics available?

Edit: Download statistics are available on quicklisp blog for January and February 2018. These suggest prove was fast gaining popularity and fiveam losing. More current statistics would be useful.

[sheepduke]

Rove should be the successor of Prove.

However I have observed a strange behavior (probably a bug) that makes tests fail randomly.

fiveAM seems to be more stable and DWIM.

[vindarel]

+1 we should at least add a warning before someone updates the recipe.

I'd be in favor of Rove. In my (short) experience, FiveAM doesn't play nice with ASDF and the host system, hence a CI system, which I expect Rove to do better. Prove did. (see https://github.com/rpgoldman/fiveam-asdf/)

[sheepduke]

I'd be in favor of Rove.

I experienced some random bugs while using Rove, in the following cases:

1. Using dexador to test HTTP server.
2. Testing functions with handler-case.

Especially the second case, it succeeds at the first time of running (asdf:test-system), but fails later.

I am in vocation and cannot provide exact code example. I will be back next week.

So I consider it as less stable compared with FiveAM.

But I am not very experienced with FiveAM, so probably I am wrong.

FiveAM doesn't play nice with ASDF

If you mean (asdf:test-system), it can be done by defining a test suite and test that suite.

[vindarel]

ACK.

It seems Prove is not archived anymore :p

[digikar99]

It seems Prove is not archived anymore :p

It isn't obsolete if it is going to be maintained, is it?

I think some of my issues with prove have more to do with my background - what would I need to study to understand the usage of pass, fail, skip, plan, finalize or to understand the ever-increasing number of tests when a test is run repeatedly:

(prove:deftest foo (is 5 5))
(prove:run-test 'foo) ; 1/1
(prove:run-test 'foo) ; 2/2
(prove:run-test 'foo) ; 3/3

May be I changed something, and I run the tests again - I'd expect the "test-numbers" would restart from 1, and not just continue. If something was broken before, and working now, this makes it not-so-easy to detect. Or may be, I'm missing the way it is supposed to be used.

What I did like about prove were the is- forms, and not seeing them elsewhere (did I overlook something?) spun up (yet another) test-lite. Even rove seems less on this. I'd be willing to fork prove (and may be maintain it) (or report and work with rove) but without the above doubts resolved, I can't seem to help. The state of testing-frameworks on cliki is already scary to say the least.

[bpseudopod]

Prove has been archived again.

[vindarel]

related #382 https://sabracrolleton.github.io/testing-framework

[vindarel]

(I'm in favor of FiveAM now too)

[digikar99]

Jeez, been a short while in the lisp world XD.

I've been using fiveam for its stability; unless I overlooked something, the comparison page has not yet benchmarked the compile and load-without-compile benchmarks of the frameworks. I felt this important in a use pattern where the tests and the code being tested are in the same system. And currently, fiveam fails miserably in this particular use case due to the use of eval - there is an open PR about this though.

[sheepduke]

By the way is FiveAM still maintained? The last commit was 13 months ago (not a big issue though) and there are open issues and PRs...

[sheepduke]

Submitted #404 for it.

[AlbertoEAF]

I used Rove more than a year ago, and it worked well. Also, it's maintained, with the latest commit this past July.

We should stick a notice that Prove was deprecated in favour of Rove until a rewrite for Rove is written.

[bpseudopod]

I second the use of Rove, on the grounds of consistency. The "getting started" article recommends cl-project for project creation, and cl-project's default project skeleton contains a Rove-based test system. See here.

[vindarel]

With your feedback OK, let's keep Prove/Rove on the page. But we'll need to know the advantages of using Rove over Fiveam. The only one I see is that… it looks more modern because it's on Github with a README.

I wouldn't say Rove is maintained. Look at the commit history: https://github.com/fukamachi/rove/commits/master The maintainer merged some simple PRs. His last fix and tag dates back from Sept, 2019. And too many issues don't have an answer: https://github.com/fukamachi/rove/issues How do you use rove from the CLI? Do tests accumulate during a lisp session without being deleted? I think I experienced this, not sure (fukamachi/rove#35).

Consistency might be a good argument. Then we can easily point to another project skeleton that uses Fiveam: https://github.com/vindarel/cl-cookieproject :)

[AlbertoEAF]

Hi @vindarel I'm going to update the Prove section for Rove, was checking it last night, it's easy to update. The API actually became smaller, but couldn't be easier to understand.

I wouldn't say Rove is maintained. ...

True, although Fukamachi has MANY cool projects and libraries I don't see him maintaining them much. It would be cool if the community could adopt and maintain them more.

[AlbertoEAF]

@vindarel:

Wow, I was testing more thoroughly Rove beyond the basic functionality and started finding some limitations in terms of design of the actual project which are not conducive of interactive development. See fukamachi/rove#49 and fukamachi/rove#50.

I now think it's really best to do a thorough and careful review of the existing unit test frameworks and make a better thought suggestion of a single one for our community. Also, if more people focus around a single test framework things will get better in the future. Eventually this might let us have all the best features of all these tools in it.

I started taking a look around the other unit test projects listed in the awesome https://github.com/CodyReichert/awesome-cl#unit-testing and found:

• CLUnit - Besides not being maintained since 2017, from the README doesn't seem to have special features to me, a bit verbose with no unique functionality not found in other test systems. Maybe I'm wrong and if we were to assess all frameworks its claims are actually unique features? Like: composable fixtures, interactive redefinition of code without changing tests (this one just seems common lisp stuff to me), interactive testing and progress report.
• FiveAM (new MR already explaining it in detail) - but apparently does not properly support fixtures
• rove - Having used it more carefully I'm starting to think it has many flaws in terms of design for larger test systems. Also, has very few test builtins. Its biggest advantage is being trivial to learn, but don't know if that's honorable as it might prevent beginners from using a more complete framework - learning one to start with, and then another when you're more experienced is (ng). It does have a very pretty output (nicer than prove - even works inside SLIME) - however, does not work well for interactive development - tests can get out of sync. Also, it has coverage, but I believe it's not rove that actually implements that but the sbcl extension sb-cover instead.
• Parachute - don't know how it actually behaves but is maintained and the API is really cool - lots of builtins like timeout, fixtures, etc. - and great tutorial - https://shinmera.github.io/parachute/. I actually believe this and FiveAM are the most well-rounded.

The remaining test libraries are for more specific purposes, but those 4 should be compared thoroughly and a rational judgement made on their merits. I think I'll try to take a look at them and see what I find. I'll try to write a review on this.

If you know of projects using any of those, or cool tutorials please tell me so I have a better understanding of how people are using them. Thank you.

[vindarel]

ah. So let's not mention Rove.

it's really best to do a thorough and careful review of the existing unit test frameworks and make a better thought suggestion of a single one for our community

FiveAM

does not properly support fixtures

it kinda does but we'll cover the mockingbird library.

Parachute - don't know how it actually behaves

I tried it and there were little things that sent me back to FiveAM. For example, the test report descriptions were less explicit by default. It doesn't have great fixtures either. I didn't find outstanding features and little things are missing (see issues). It doesn't have some 5AM features (random testing, maybe more).

I'll try to write a review on this.

see https://sabracrolleton.github.io/testing-framework#org2ebb74b

[sheepduke]

does not properly support fixtures

it kinda does but we'll cover the mockingbird library.

To be honest, I am not really sure if it is fair to say so, since I recently saw extensive usage of it in cl-gserver project. I think we need to re-evaluate FiveAM's fixture feature.

My personal experience with Rove was that I eventually found some limitations and finally went to FiveAM, which was more stable and had less surprises.

But since my experience was 2 years ago, it might be outdated...

[vindarel]

5AM's fixtures are like a macro. Mockingbird allows to answer more questions, like "how many times was this function called?". This is required in a good test setup.

In cl-gserver he uses cl-mock: https://github.com/mdbergmann/cl-gserver/blob/b000aa8d849fafc9994200393059088ea1a2b22a/tests/router-test.lisp

Quick discussion about mockingbird vs cl-mock: CodyReichert/awesome-cl#392

[sheepduke]

5AM's fixtures are like a macro. Mockingbird allows to answer more questions, like "how many times was this function called?". This is required in a good test setup.

In cl-gserver he uses cl-mock: https://github.com/mdbergmann/cl-gserver/blob/b000aa8d849fafc9994200393059088ea1a2b22a/tests/router-test.lisp

Quick discussion about mockingbird vs cl-mock: CodyReichert/awesome-cl#392

True. Mockingbird will be added anyway...

[sheepduke]

@vindarel Sorry for bringing this up again.

I made a mistake about cl-gserver using Mockingbird. You are right that cl-mock is used there.

We may further discuss mockingbird vs cl-mock in that thread.