Add a way to export failing test cases

[dzamlo]

It would be very convenient to add a way to export failing test cases to allow to reproduce them and make a regression test suite.

The failing test should be exported either as:

• a test function (with the #[test] attribute) you can just copy-paste inside your code or
• a file containing the test cases with a way to rerun these test case automatically

This feature is more or less presented in the following video:
https://www.youtube.com/watch?v=x7O2Hkq983Y
And is used on:
http://www.quickcheck-ci.com/

[BurntSushi]

This would be pretty neat.

Exporting actual Rust code seems like it'd be difficult, so I think the more reasonable path here is exporting the arguments given to a QuickCheck property that caused it to fail. (It's already doing this to some extent, but it's not machine readable and there's no way to "run tests with these arguments.")

[milibopp]

I think, a first step towards this would be to implement a function similar to quickcheck::quickcheck, which in addition takes a list of test values that must always be checked, no matter what values get generated. Then one could add failing test cases as hard-coded arguments to this function.

[Eh2406]

This is some pseudocode to make a wrapper that dumps failing cases to a folder. With some help it could work like Hypothesis's caching of failed test.

extern crate quickcheck;
use quickcheck::{Arbitrary, Gen};

extern crate serde;
use serde::{Serialize, Deserialize};
extern crate serde_json;

use std::error::Error;
use std::io::prelude::*;
use std::fs::File;
use std::path::Path;

#[derive(Debug, Clone)]
pub struct SavedCheck<T: Clone + Send> {
    iner: T,
}

impl<T: Arbitrary + Serialize + Deserialize> Arbitrary for SavedCheck<T> {
    fn arbitrary<G: Gen>(g: &mut G) -> Self {
        for file in Path::new("quickcheck_cache/") {
            let file = File::open(&file);
            if let Ok(t) = serde_json::from_str(&file.as_bytes()) {
                // somehow only return each one once
                return SavedCheck { iner: t };
            }
        }
        SavedCheck { iner: Arbitrary::arbitrary(g) }
    }

    fn shrink(&self) -> Box<Iterator<Item = Self>> {
        if let Ok(st) = serde_json::to_string(&self.iner) {
            // somehow pick a unique name
            let path = Path::new("quickcheck_cache/test.json");
            let display = path.display();
            let mut file = match File::create(&path) {
                Err(why) => panic!("couldn't create {}: {}", display, why.description()),
                Ok(file) => file,
            };
            match file.write_all(st.as_bytes()) {
                Err(why) => panic!("couldn't write to {}: {}", display, why.description()),
                Ok(_) => println!("successfully wrote to {}", display),
            }
        }
        Box::new(self.iner.shrink().map(|t| SavedCheck { iner: t }))
    }
}

[BurntSushi]

It sounds like proptest can do this. I doubt I'll have the bandwidth to implement/review/maintain something like this.