Skip to content

Trials

Jump to bottom
Martin Pool edited this page Mar 3, 2024 · 13 revisions

cargo-mutants

Can we get cargo-mutants itself clean?

2022-09-11

There are a few failures:

src/cargo.rs:138: replace terminate_child -> Result<()> with Ok(Default::default()) ... TIMEOUT in 2.7s build + 44.4s test
src/cargo.rs:162: replace terminate_child -> Result<()> with Ok(Default::default()) ... NOT CAUGHT in 2.0s build + 8.5s test
src/cargo.rs:176: replace setpgid_on_unix -> PopenConfig with Default::default() ... TIMEOUT in 2.1s build + 44.4s test
src/cargo.rs:184: replace setpgid_on_unix -> PopenConfig with Default::default() ... NOT CAUGHT in 2.2s build + 8.5s test
src/interrupt.rs:14: replace install_handler with () ... NOT CAUGHT in 2.6s build + 8.2s test
src/interrupt.rs:20: replace check_interrupted -> Result<()> with Ok(Default::default()) ... NOT CAUGHT in 2.1s build + 8.4s test
src/lab.rs:141: replace minimum_test_timeout -> Result<Duration> with Ok(Default::default()) ... NOT CAUGHT in 2.4s build + 8.7s test
src/log_file.rs:73: replace last_line -> Result<String> with Ok(Default::default()) ... NOT CAUGHT in 2.2s build + 8.5s test
src/options.rs:67: replace Options::has_test_timeout -> bool with false ... NOT CAUGHT in 2.0s build + 23.6s test
src/options.rs:71: replace Options::set_test_timeout with () ... NOT CAUGHT in 1.9s build + 8.2s test
src/outcome.rs:219: replace ScenarioOutcome::check_or_build_failed -> bool with false ... NOT CAUGHT in 2.1s build + 8.2s test
src/source.rs:222: replace should_mutate_target -> bool with true ... NOT CAUGHT in 2.0s build + 8.2s test
src/visit.rs:269: replace attr_is_mutants_skip::list_is_mutants_skip -> bool with true ... NOT CAUGHT in 2.3s build + 8.5s test
tests/cli.rs:1130: replace check_text_list_output with () ... NOT CAUGHT in 2.0s build + 8.4s test
tests/cli.rs:41: replace <impl CommandInstaExt for std::process::Command>::assert_insta with () ... NOT CAUGHT in 2.1s build + 8.1s test
167 mutants tested in 17:06: 13 missed, 117 caught, 35 unviable, 2 timeouts

There was a bug that non-test functions in tests/ were being mutated, which is not helpful. That's now fixed and will be released soon.

The failures about terminate_child and setpgid_on_unix partly show that cargo-mutants does not yet have a good concept of mutating functions only on some platforms that can test them. https://github.com/sourcefrog/cargo-mutants/issues/50. For now I just made the Windows setpgid_on_unix skip, since it does nothing anyhow. The other one may be harder.

The timeouts on terminate_child seem to show a real shortcoming in the tests: if cargo-mutants hangs, the tests will hang as well. assert_fs has a handy timeout method we could use...

I debugged this by manually inserting a similar bug into the source and then trying to get the tests to fail as expected.

There were some missed mutants around set_test_timeout and has_test_timeout. I decided these were really indicators that this code could be better-factored: these low-value methods aren't adding much value but also we shouldn't mutate the Options structure. I refactored to remove them and to explicitly pass a test_timeout argument to some functions. This ends up with a net deletion of code.

I realized that some tests were falsely passing because my environment had INSTA_UPDATE=always for https://insta.rs, meaning that any tests that depend only on Insta snapshots would pass! Not good. With that fixed, the build_globset and probably other failures will go away.

There are two missed functions in interrupts.rs, both to do with how cargo-mutants catchs and responds to Ctrl-C. One can be handled with a test that it does notice a Ctrl-C and terminates the cargo child.

check_interrupted is more complex; if it returns false then the program will never notice that it was interrupted: a classic case of a mutation that causes a hang. For now I've just marked it mutants::skip. This category could actually be specifically handled with an attribute that means: this function may cause a hang. This could normally be skipped, but we could also try the mutation and check that it is either caught or times out.

2024-03-03

https://github.com/sourcefrog/cargo-mutants/blob/939fbfaadd29a8aa8e0e36b7432dc1c84fe51533/src/lab.rs#L222-L248

It complains that the == in line 246 isn't caught... and on closer inspection, this condition to exit the loop early is entirely redundant, because when options.check_only then the only phase is to check! Nice!

fs\_at

2022-08-05, https://github.com/rbtcollins/fs_at 21a7b61c9dbdfdf890fc7d556e04d9150fc3aaf9 using cargo-mutants 0.2.10-pre.

It worked first try, and it finds some gaps, but they're all in Windows implementations, when I'm testing on Linux.

https://gist.github.com/7ec27bceea3fd27cf729310e195ff9b8

This was basically a success showing the crate's well-tested, at least on Linux.

Detecting to skip the Windows modules automatically would have been ideal, although that's complicated a bit by the fact that it uses cfg_if not simple cfg attrs. (See https://github.com/rbtcollins/fs_at/pull/8.)

Just adding #[mutants::skip] into the Windows modules would make it succeed on Linux but then mutation testing would not be useful on Windows.

Conclusions:

time

2022-08-05, https://github.com/time-rs/time using cargo-mutants 0.2.10-pre.

This failed tests in the clean tree, and when I investigated I saw that it also fails in a clean checkout. It turns out (https://github.com/time-rs/time/issues/497) that the tests fail unless you use cargo test --all-features.

I tried cargo mutants -- --all-features, but it still failed, because the extra arguments to cargo mutants are passed only to cargo test, and this needs to go to cargo build --tests as well, or the build fails.

It seems a bit unfriendly to new contributors that cargo test won't pass by default, but it's not unreasonable.

In any case there's a general issue about supporting crates with features: people might want to test mutations with non-default features on.

Conclusions:

2022-08-07

With cargo mutants -C --all-features, it works, finding 678 mutants, some of which seem to be missed in interesting ways.

; cargo r --release mutants -d ~/src/time  -C --all-features
    Finished release [optimized] target(s) in 0.07s
     Running `target/release/cargo-mutants mutants -d /home/mbp/src/time -C --all-features`
Freshen source tree ... ok in 0.051s
Copy source and build products to scratch directory ... 1380 MB in 2.176s
Unmutated baseline ... ok in 28.726s
Auto-set test timeout to 143.5s
Found 678 mutants to test
src/parsing/parsed.rs:392: replace Parsed::set_leap_second_allowed with () ... NOT CAUGHT in 19.561s
src/sys/local_offset_at/unix.rs:48: replace tm_to_offset -> Option<UtcOffset> with Default::default() ... NOT CAUGHT in 19.102s
src/duration.rs:45: replace <impl Debug for Duration>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 18.326s
src/sys/local_offset_at/windows.rs:57: replace filetime_to_secs -> i64 with Default::default() ... NOT CAUGHT in 17.548s
src/sys/local_offset_at/windows.rs:79: replace local_offset_at -> Option<UtcOffset> with Default::default() ... NOT CAUGHT in 12.859s
src/error/invalid_variant.rs:11: replace <impl Display for InvalidVariant>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 17.885s
interrupted
Error: interrupted

imbl

2022-08-05, https://github.com/jneem/imbl using cargo-mutants 0.2.10-pre. Testing v2.0.0-1-g8756e76, 8756e765d8f76688da333b751db14287ae862c20.

I was going to test <https://github.com/bodil/im-rs but it seems that has been abandoned and forked as imbl, so I'll do imbl instead.

Seems like it should be hermetic, and it has no workspace, so 🤞.

It finds 830 mutants and on my laptop is predicting it'll take three hours to try them all. But it does seem to be proceeding and finding some gaps. Some look like they might be real test coverage gaps.

Partial output: https://gist.github.com/sourcefrog/5475f9695e4a42e97c19551d1d79f42b

; nice cargo mutants -d ~/src/imbl
Freshen source tree ... ok in 14.316s
Copy source and build products to scratch directory ... 559 MB in 0.489s
Unmutated baseline ... ok in 15.745s
Auto-set test timeout to 78.5s
Found 836 mutants to test
src/vector/mod.rs:1904: replace <impl Sum for Vector<A>>::sum -> Self with Default::default() ... NOT CAUGHT in 22.992s
src/hash/set.rs:939: replace <impl From for HashSet<A, S>>::from -> Self with Default::default() ... NOT CAUGHT in 17.512s
src/ord/set.rs:113: replace <impl BTreeValue for Value<A>>::ptr_eq -> bool with true ... NOT CAUGHT in 17.951s
src/ord/map.rs:1220: replace OrdMap<K, V>::symmetric_difference_with -> Self with Default::default() ... NOT CAUGHT in 18.657s
src/hash/map.rs:1834: replace <impl Iterator for IterMut<'a, K, V>>::next -> Option<Self::Item> with Default::default() ... NOT CAUGHT in 18.366s
src/ord/set.rs:272: replace OrdSet<A>::ptr_eq -> bool with false ... NOT CAUGHT in 21.529s
src/ord/set.rs:487: replace OrdSet<A>::insert -> Option<A> with Default::default() ... TIMEOUT in 84.335s
src/tests/mod.rs:5: replace code_fmt -> String with "".into() ... NOT CAUGHT in 20.769s
src/hash/set.rs:796: replace <impl Debug for HashSet<A, S>>::fmt -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 17.620s
interrupted
Error: interrupted

Freshen source tree ... ok in 0.033s
Copy source and build products to scratch directory ... 559 MB in 0.448s
Unmutated baseline ... ok in 15.631s
Auto-set test timeout to 77.9s
Found 836 mutants to test
src/hash/map.rs:116: replace <impl HashValue for (K, V)>::ptr_eq -> bool with true ... NOT CAUGHT in 18.191s
src/hash/set.rs:919: replace <impl From for HashSet<OA, SB>>::from -> Self with Default::default() ... NOT CAUGHT in 18.720s
src/vector/mod.rs:1941: replace <impl Extend for Vector<A>>::extend with () ... NOT CAUGHT in 20.633s
src/hash/set.rs:534: replace HashSet<A, S>::unions -> Self with Default::default() ... NOT CAUGHT in 20.982s
src/ord/set.rs:1145: replace <impl From for OrdSet<A>>::from -> Self with Default::default() ... NOT CAUGHT in 19.004s
src/vector/mod.rs:706: replace Vector<A>::dot -> std::io::Result<()> with Ok(Default::default()) ... NOT CAUGHT in 21.798s
src/hash/map.rs:221: replace HashMap<K, V, S>::ptr_eq -> bool with true ... NOT CAUGHT in 21.155s
src/hash/map.rs:2036: replace <impl From for HashMap<K, V, S>>::from -> Self with Default::default() ... NOT CAUGHT in 18.540s
src/ord/set.rs:152: replace <impl BTreeValue for Value<A>>::search_value -> Result<usize, usize> with Ok(Default::default()) ... NOT CAUGHT in 21.170s
src/nodes/rrb.rs:405: replace Node<A>::size -> usize with Default::default() ... NOT CAUGHT in 19.158s
src/vector/mod.rs:1842: replace <impl PartialEq for Vector<A>>::eq::cmp_chunk -> bool with false ... NOT CAUGHT in 20.426s
src/ord/map.rs:1815: replace <impl Extend for OrdMap<K, V>>::extend with () ... NOT CAUGHT in 21.744s
src/vector/rayon.rs:74: replace <impl IndexedParallelIterator for ParIter<'a, A>>::len -> usize with Default::default() ... NOT CAUGHT in 20.734s
src/ord/set.rs:972: replace <impl Iterator for Iter<'a, A>>::size_hint -> (usize, Option<usize>) with Default::default() ... NOT CAUGHT in 17.709s
src/ord/map.rs:1481: replace OrdMap<K, V>::without_max -> (Option<V>, Self) with Default::default() ... NOT CAUGHT in 22.209s
src/ser.rs:49: replace <impl Visitor for SeqVisitor<'de, S, A>>::expecting -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 20.054s
interrupted
Error: interrupted

Conclusions:

  • Let it finish running (on a bigger machine)
  • Look in detail at some more of the missed mutations and see if they really seem to be coverage gaps. Report here and maybe offer the report to the maintainers.
  • The experiments often end up using only a single core, so this is another case where running multiple mutants in parallel would be useful https://github.com/sourcefrog/cargo-mutants/issues/39.

apollo-rs

2022-08-06 https://github.com/apollographql/apollo-rs

The repo is a workspace with no root package, and cargo-mutants doesn't support that yet.

Conclusions:

tar-rs

https://github.com/alexcrichton/tar-rs

2022-08-06, cargo-mutants v0.2.10-pre, 0.4.38-2-gf4f439c, f4f439c

A nice fast test suite that takes about 2s on my laptop. Seems like it should be hermetic and well tested. Finds 281 mutants.

Freshen source tree ... ok in 2.877s
Copy source and build products to scratch directory ... 83 MB in 0.099s
Unmutated baseline ... ok in 2.024s
Auto-set test timeout to 20.0s
Found 281 mutants to test
281 mutants tested in 9:26: 234 caught, 47 unviable

Conclusions:

  • Everything was caught! An outstanding result.
  • We could perhaps test this from CI on either cargo-mutants or tar-rs.

toml-rs

https://github.com/alexcrichton/toml-rs

2022-08-06, cargo-mutants v0.2.10-pre, 0.5.9-2-gcf86793, cf86793.

This also seems like something that would be amenable to good hermetic tests. But, it seems like there are what look like true-positive gaps, and also, interestingly, what seems like a hang:

src/tokens.rs:250: replace Tokenizer<'a>::eatc -> bool with true ... TIMEOUT in 21.004s

That's not too surprising given the name, since it probably governs a loop. It's nice that the timeout code seems to have detected and interrupted it.

There's a long list of findings:

Freshen source tree ... ok in 5.669s
Copy source and build products to scratch directory ... 191 MB in 0.165s
Unmutated baseline ... ok in 1.300s
Auto-set test timeout to 20.0s
Found 481 mutants to test
src/value.rs:148: replace Value::is_str -> bool with true ... NOT CAUGHT in 2.843s
src/map.rs:72: replace Map<String, Value>::clear with () ... NOT CAUGHT in 1.880s
src/ser.rs:862: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_unit -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.707s
src/map.rs:288: replace <impl Deserialize for Map<String, Value>>::deserialize::<impl Visitor for Visitor>::expecting -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 1.770s
src/ser.rs:851: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_none -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.720s
src/value.rs:148: replace Value::is_str -> bool with false ... NOT CAUGHT in 2.020s
src/value.rs:135: replace Value::is_bool -> bool with false ... NOT CAUGHT in 2.008s
src/de.rs:335: replace build_table_indices -> HashMap<Vec<Cow<'de, str>>, Vec<usize>> with Default::default() ... NOT CAUGHT in 2.394s
src/map.rs:282: replace <impl Deserialize for Map<String, Value>>::deserialize -> Result<Self, D::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.362s
src/value.rs:122: replace Value::is_float -> bool with false ... NOT CAUGHT in 2.222s
src/map.rs:223: replace <impl PartialEq for Map<String, Value>>::eq -> bool with true ... NOT CAUGHT in 1.973s
src/de.rs:1046: replace <impl MapAccess for DatetimeDeserializer<'de>>::next_key_seed -> Result<Option<K::Value>, Error> with Ok(Default::default()) ... NOT CAUGHT in 2.290s
src/value.rs:682: replace <impl MapAccess for MapDeserializer>::next_key_seed -> Result<Option<T::Value>, crate::de::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.888s
src/de.rs:2028: replace Error::line_col -> Option<(usize, usize)> with Default::default() ... NOT CAUGHT in 2.190s
src/ser.rs:1234: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_u32 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.056s
src/ser.rs:1429: replace <impl Serializer for StringExtractor>::serialize_bytes -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.739s
src/ser.rs:1385: replace <impl Serializer for StringExtractor>::serialize_i16 -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.738s
src/ser.rs:1433: replace <impl Serializer for StringExtractor>::serialize_none -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.688s
src/ser.rs:858: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_some -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.706s
src/spanned.rs:77: replace <impl Borrow for Spanned<String>>::borrow -> &str with Default::default() ... NOT CAUGHT in 1.738s
src/ser.rs:1417: replace <impl Serializer for StringExtractor>::serialize_f64 -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.730s
src/map.rs:64: replace Map<String, Value>::with_capacity -> Self with Default::default() ... NOT CAUGHT in 1.807s
src/tokens.rs:250: replace Tokenizer<'a>::eatc -> bool with true ... TIMEOUT in 21.004s
src/value.rs:181: replace Value::as_array_mut -> Option<&mut Vec<Value>> with Default::default() ... NOT CAUGHT in 2.558s
src/value.rs:210: replace Value::is_table -> bool with true ... NOT CAUGHT in 2.319s
src/value.rs:329: replace <impl Index for usize>::index -> Option<&'a Value> with Default::default() ... NOT CAUGHT in 2.395s
src/value.rs:114: replace Value::as_float -> Option<f64> with Default::default() ... NOT CAUGHT in 1.855s
src/tokens.rs:229: replace Tokenizer<'a>::eat_newline_or_eof -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 2.216s
src/ser.rs:846: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_bytes -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.139s
src/ser.rs:1206: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_bool -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.265s
src/ser.rs:1444: replace <impl Serializer for StringExtractor>::serialize_unit -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.856s
src/map.rs:160: replace Map<String, Value>::len -> usize with Default::default() ... NOT CAUGHT in 1.722s
src/ser.rs:1468: replace <impl Serializer for StringExtractor>::serialize_newtype_struct -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.788s
src/spanned.rs:56: replace Spanned<T>::span -> (usize, usize) with Default::default() ... NOT CAUGHT in 1.702s
src/de.rs:2065: replace Error::fix_offset with () ... NOT CAUGHT in 2.192s
src/ser.rs:161: replace ArraySettings::pretty -> ArraySettings with Default::default() ... NOT CAUGHT in 2.291s
src/value.rs:970: replace <impl SerializeTupleVariant for SerializeVec>::serialize_field -> Result<(), crate::ser::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.339s
src/ser.rs:1405: replace <impl Serializer for StringExtractor>::serialize_u32 -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.906s
src/macros.rs:413: replace insert_toml with () ... NOT CAUGHT in 1.786s
src/map.rs:256: replace <impl Debug for Map<String, Value>>::fmt -> Result<(), fmt::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.936s
src/ser.rs:1144: replace <impl SerializeStruct for SerializeTable<'a, 'b>>::serialize_field -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 2.121s
src/ser.rs:837: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_str -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.063s
src/map.rs:137: replace Map<String, Value>::remove -> Option<Value> with Default::default() ... NOT CAUGHT in 2.043s
src/ser.rs:1274: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_unit -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.906s
src/ser.rs:1246: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_f64 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.305s
src/ser.rs:1214: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_i16 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.873s
src/ser.rs:899: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_newtype_variant -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.709s
src/de.rs:1933: replace Deserializer<'a>::eat_comment -> Result<bool, Error> with Ok(Default::default()) ... NOT CAUGHT in 2.227s
src/de.rs:2058: replace Error::add_key_context with () ... NOT CAUGHT in 2.154s
src/de.rs:1302: replace Deserializer<'a>::set_allow_duplicate_after_longer_table with () ... NOT CAUGHT in 1.962s
src/value.rs:168: replace Value::is_datetime -> bool with true ... NOT CAUGHT in 2.207s
src/de.rs:548: replace <impl SeqAccess for MapVisitor<'de, 'b>>::next_element_seed -> Result<Option<K::Value>, Error> with Ok(Default::default()) ... NOT CAUGHT in 2.657s
src/map.rs:98: replace Map<String, Value>::contains_key -> bool with false ... NOT CAUGHT in 2.507s
src/value.rs:938: replace <impl SerializeTuple for SerializeVec>::serialize_element -> Result<(), crate::ser::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.206s
src/ser.rs:1421: replace <impl Serializer for StringExtractor>::serialize_char -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.944s
src/ser.rs:808: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_u8 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.936s
src/ser.rs:45: replace to_vec -> Result<Vec<u8>, Error> with Ok(Default::default()) ... NOT CAUGHT in 1.936s
src/de.rs:1937: replace Deserializer<'a>::eat_newline_or_eof -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 2.338s
src/value.rs:706: replace <impl MapAccess for MapDeserializer>::size_hint -> Option<usize> with Default::default() ... NOT CAUGHT in 2.039s
src/value.rs:215: replace Value::same_type -> bool with false ... NOT CAUGHT in 2.320s
src/map.rs:166: replace Map<String, Value>::is_empty -> bool with true ... NOT CAUGHT in 1.738s
src/value.rs:921: replace <impl SerializeSeq for SerializeVec>::serialize_element -> Result<(), crate::ser::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.817s
src/ser.rs:820: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_u64 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.057s
src/ser.rs:1093: replace <impl SerializeMap for SerializeTable<'a, 'b>>::serialize_value -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 1.988s
src/tokens.rs:456: replace Tokenizer<'a>::substr_offset -> usize with Default::default() ... NOT CAUGHT in 1.958s
src/value.rs:1054: replace <impl Visitor for DatetimeOrTable<'a>>::expecting -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 1.890s
src/value.rs:168: replace Value::is_datetime -> bool with false ... NOT CAUGHT in 2.008s
src/ser.rs:1481: replace <impl Serializer for StringExtractor>::serialize_newtype_variant -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.039s
src/ser.rs:1389: replace <impl Serializer for StringExtractor>::serialize_i32 -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.808s
src/ser.rs:730: replace Serializer<'a>::emit_key_part -> Result<bool, Error> with Ok(Default::default()) ... NOT CAUGHT in 2.248s
src/ser.rs:1035: replace <impl SerializeTuple for SerializeSeq<'a, 'b>>::end -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 2.171s
src/ser.rs:1780: replace <impl SerializeSeq for Categorize<E>>::serialize_element -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.293s
src/de.rs:2076: replace Error::fix_linecol with () ... NOT CAUGHT in 2.223s
src/ser.rs:1079: replace <impl SerializeMap for SerializeTable<'a, 'b>>::serialize_key -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 2.288s
src/spanned.rs:83: replace <impl PartialEq for Spanned<T>>::eq -> bool with true ... NOT CAUGHT in 2.219s
src/map.rs:323: replace <impl FromIterator for Map<String, Value>>::from_iter -> Self with Default::default() ... NOT CAUGHT in 1.725s
src/value.rs:96: replace Value::get_mut -> Option<&mut Value> with Default::default() ... NOT CAUGHT in 2.423s
src/ser.rs:525: replace Serializer<'a>::emit_str -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 2.312s
src/value.rs:361: replace <impl Index for String>::index -> Option<&'a Value> with Default::default() ... NOT CAUGHT in 2.311s
src/ser.rs:792: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_i8 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.726s
src/ser.rs:1270: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_some -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.723s
src/spanned.rs:124: replace <impl Deserialize for Spanned<T>>::deserialize::<impl Visitor for SpannedVisitor<T>>::expecting -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 1.708s
src/de.rs:1292: replace Deserializer<'a>::set_require_newline_after_table with () ... NOT CAUGHT in 1.765s
src/ser.rs:828: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_f64 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.056s
src/ser.rs:1457: replace <impl Serializer for StringExtractor>::serialize_unit_variant -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.060s
src/value.rs:1073: replace <impl Visitor for DatetimeOrTable<'a>>::visit_string -> Result<bool, E> with Ok(Default::default()) ... NOT CAUGHT in 1.739s
src/datetime.rs:420: replace digit -> Result<u8, DatetimeParseError> with Ok(Default::default()) ... NOT CAUGHT in 1.829s
src/de.rs:2250: replace E<'a>::type_name -> &'static str with Default::default() ... NOT CAUGHT in 1.824s
src/datetime.rs:217: replace <impl Display for Time>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 2.227s
src/value.rs:109: replace Value::is_integer -> bool with false ... NOT CAUGHT in 2.157s
src/macros.rs:419: replace push_toml with () ... NOT CAUGHT in 2.023s
src/datetime.rs:211: replace <impl Display for Date>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 1.761s
src/value.rs:954: replace <impl SerializeTupleStruct for SerializeVec>::serialize_field -> Result<(), crate::ser::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.736s
src/value.rs:101: replace Value::as_integer -> Option<i64> with Default::default() ... NOT CAUGHT in 1.957s
src/ser.rs:1852: replace <impl SerializeStruct for Categorize<E>>::serialize_field -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.989s
src/map.rs:304: replace <impl Deserialize for Map<String, Value>>::deserialize::<impl Visitor for Visitor>::visit_map -> Result<Self::Value, V::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.813s
src/ser.rs:1413: replace <impl Serializer for StringExtractor>::serialize_f32 -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.845s
src/ser.rs:1238: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_u64 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.823s
src/ser.rs:1047: replace <impl SerializeTupleVariant for SerializeSeq<'a, 'b>>::serialize_field -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 1.806s
src/ser.rs:866: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_unit_struct -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.316s
src/ser.rs:1440: replace <impl Serializer for StringExtractor>::serialize_some -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.770s
src/ser.rs:477: replace Serializer<'a>::emit_array -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 1.958s
src/ser.rs:886: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_newtype_struct -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.920s
src/spanned.rs:97: replace <impl PartialOrd for Spanned<T>>::partial_cmp -> Option<Ordering> with Default::default() ... NOT CAUGHT in 1.806s
src/value.rs:173: replace Value::as_array -> Option<&Vec<Value>> with Default::default() ... NOT CAUGHT in 1.946s
src/ser.rs:1836: replace <impl SerializeMap for Categorize<E>>::serialize_value -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.009s
src/value.rs:647: replace <impl SeqAccess for SeqDeserializer>::next_element_seed -> Result<Option<T::Value>, crate::de::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.892s
src/ser.rs:1051: replace <impl SerializeTupleVariant for SerializeSeq<'a, 'b>>::end -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 2.374s
src/tokens.rs:268: replace Tokenizer<'a>::input -> &'a str with Default::default() ... NOT CAUGHT in 1.753s
src/datetime.rs:484: replace <impl Deserialize for DatetimeKey>::deserialize::<impl Visitor for FieldVisitor>::expecting -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 1.769s
src/value.rs:109: replace Value::is_integer -> bool with true ... NOT CAUGHT in 2.058s
src/ser.rs:1242: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_f32 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.299s
src/ser.rs:1218: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_i32 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.720s
src/ser.rs:666: replace Serializer<'a>::emit_table_header -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 2.209s
src/value.rs:365: replace <impl Index for String>::index_mut -> Option<&'a mut Value> with Default::default() ... NOT CAUGHT in 2.220s
src/ser.rs:1377: replace <impl Serializer for StringExtractor>::serialize_bool -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.754s
src/ser.rs:1278: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_unit_struct -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.706s
src/ser.rs:1250: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_char -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.161s
src/map.rs:334: replace <impl Extend for Map<String, Value>>::extend with () ... NOT CAUGHT in 1.721s
src/ser.rs:796: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_i16 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.047s
src/map.rs:53: replace Map<String, Value>::with_capacity -> Self with Default::default() ... NOT CAUGHT in 2.104s
src/de.rs:1672: replace Deserializer<'a>::float -> Result<f64, Error> with Ok(Default::default()) ... NOT CAUGHT in 2.391s
src/value.rs:135: replace Value::is_bool -> bool with true ... NOT CAUGHT in 2.156s
src/value.rs:378: replace <impl Index for &'s T>::index_mut -> Option<&'a mut Value> with Default::default() ... NOT CAUGHT in 2.109s
src/datetime.rs:228: replace <impl Display for Offset>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 1.979s
src/ser.rs:1425: replace <impl Serializer for StringExtractor>::serialize_str -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.914s
src/ser.rs:1210: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_i8 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.756s
src/ser.rs:1448: replace <impl Serializer for StringExtractor>::serialize_unit_struct -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.769s
src/ser.rs:982: replace <impl SerializeSeq for SerializeSeq<'a, 'b>>::serialize_element -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 1.931s
src/tokens.rs:535: replace Token<'a>::describe -> &'static str with Default::default() ... NOT CAUGHT in 1.740s
src/ser.rs:1031: replace <impl SerializeTuple for SerializeSeq<'a, 'b>>::serialize_element -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 1.730s
src/map.rs:166: replace Map<String, Value>::is_empty -> bool with false ... NOT CAUGHT in 1.756s
src/value.rs:160: replace Value::as_datetime -> Option<&Datetime> with Default::default() ... NOT CAUGHT in 2.052s
src/ser.rs:1397: replace <impl Serializer for StringExtractor>::serialize_u8 -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.128s
src/ser.rs:442: replace Serializer<'a>::_emit_key -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 2.019s
src/ser.rs:1819: replace <impl SerializeTupleStruct for Categorize<E>>::serialize_field -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.971s
src/value.rs:220: replace Value::type_str -> &'static str with Default::default() ... NOT CAUGHT in 1.979s
src/ser.rs:875: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_unit_variant -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.053s
src/map.rs:296: replace <impl Deserialize for Map<String, Value>>::deserialize::<impl Visitor for Visitor>::visit_unit -> Result<Self::Value, E> with Ok(Default::default()) ... NOT CAUGHT in 2.024s
src/ser.rs:1806: replace <impl SerializeTupleVariant for Categorize<E>>::serialize_field -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.687s
src/ser.rs:1298: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_newtype_struct -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.873s
src/value.rs:189: replace Value::is_array -> bool with false ... NOT CAUGHT in 2.023s
src/ser.rs:788: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_bool -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.008s
src/ser.rs:103: replace to_string_pretty -> Result<String, Error> with Ok(Default::default()) ... NOT CAUGHT in 1.927s
src/ser.rs:1287: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_unit_variant -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.925s
src/ser.rs:997: replace <impl SerializeSeq for SerializeSeq<'a, 'b>>::end -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 1.825s
src/ser.rs:1067: replace <impl SerializeTupleStruct for SerializeSeq<'a, 'b>>::end -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 2.241s
src/ser.rs:1535: replace <impl Display for Error>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 2.373s
src/datetime.rs:539: replace <impl Display for DatetimeParseError>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 1.990s
src/ser.rs:1393: replace <impl Serializer for StringExtractor>::serialize_i64 -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.820s
src/ser.rs:1179: replace <impl SerializeStruct for SerializeTable<'a, 'b>>::end -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 1.889s
src/ser.rs:1226: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_u8 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.889s
src/map.rs:223: replace <impl PartialEq for Map<String, Value>>::eq -> bool with false ... NOT CAUGHT in 1.710s
src/ser.rs:816: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_u32 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.759s
src/ser.rs:1401: replace <impl Serializer for StringExtractor>::serialize_u16 -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.752s
src/tokens.rs:221: replace Tokenizer<'a>::eat_comment -> Result<bool, Error> with Ok(Default::default()) ... NOT CAUGHT in 2.190s
src/datetime.rs:450: replace <impl Deserialize for Datetime>::deserialize::<impl Visitor for DatetimeVisitor>::expecting -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 2.110s
src/datetime.rs:519: replace <impl Deserialize for DatetimeFromString>::deserialize::<impl Visitor for Visitor>::expecting -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 1.758s
src/value.rs:215: replace Value::same_type -> bool with true ... NOT CAUGHT in 1.716s
src/de.rs:2009: replace Deserializer<'a>::to_linecol -> (usize, usize) with Default::default() ... NOT CAUGHT in 2.209s
src/value.rs:654: replace <impl SeqAccess for SeqDeserializer>::size_hint -> Option<usize> with Default::default() ... NOT CAUGHT in 2.275s
src/ser.rs:1222: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_i64 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.868s
src/spanned.rs:83: replace <impl PartialEq for Spanned<T>>::eq -> bool with false ... NOT CAUGHT in 1.906s
src/ser.rs:1409: replace <impl Serializer for StringExtractor>::serialize_u64 -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.722s
src/de.rs:1282: replace Deserializer<'a>::end -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 1.819s
src/ser.rs:1263: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_none -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.812s
src/ser.rs:1832: replace <impl SerializeMap for Categorize<E>>::serialize_key -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.689s
src/value.rs:384: replace <impl Display for Value>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 2.224s
src/value.rs:210: replace Value::is_table -> bool with false ... NOT CAUGHT in 2.221s
src/value.rs:127: replace Value::as_bool -> Option<bool> with Default::default() ... NOT CAUGHT in 2.190s
src/value.rs:122: replace Value::is_float -> bool with true ... NOT CAUGHT in 2.190s
src/value.rs:986: replace <impl SerializeMap for SerializeMap>::serialize_key -> Result<(), crate::ser::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.120s
src/spanned.rs:91: replace <impl Hash for Spanned<T>>::hash with () ... NOT CAUGHT in 1.837s
src/ser.rs:800: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_i32 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.041s
src/ser.rs:1381: replace <impl Serializer for StringExtractor>::serialize_i8 -> Result<String, Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.023s
src/value.rs:352: replace <impl Index for str>::index_mut -> Option<&'a mut Value> with Default::default() ... NOT CAUGHT in 1.952s
src/ser.rs:90: replace to_string -> Result<String, Error> with Ok(Default::default()) ... NOT CAUGHT in 2.826s
src/ser.rs:511: replace Serializer<'a>::escape_key -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 3.172s
src/ser.rs:1230: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_u16 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.489s
src/ser.rs:832: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_char -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.089s
src/ser.rs:824: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_f32 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 2.205s
src/de.rs:372: replace headers_equal -> bool with false ... NOT CAUGHT in 2.374s
src/datetime.rs:491: replace <impl Deserialize for DatetimeKey>::deserialize::<impl Visitor for FieldVisitor>::visit_str -> Result<(), E> with Ok(Default::default()) ... NOT CAUGHT in 2.539s
src/ser.rs:1311: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_newtype_variant -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.839s
src/value.rs:455: replace <impl Deserialize for Value>::deserialize::<impl Visitor for ValueVisitor>::expecting -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 1.723s
src/ser.rs:1793: replace <impl SerializeTuple for Categorize<E>>::serialize_element -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.752s
src/ser.rs:1259: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_bytes -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.688s
src/datetime.rs:187: replace <impl Debug for Datetime>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 1.706s
src/ser.rs:1063: replace <impl SerializeTupleStruct for SerializeSeq<'a, 'b>>::serialize_field -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 1.707s
src/value.rs:997: replace <impl SerializeMap for SerializeMap>::serialize_value -> Result<(), crate::ser::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.807s
src/value.rs:336: replace <impl Index for usize>::index_mut -> Option<&'a mut Value> with Default::default() ... NOT CAUGHT in 1.922s
src/ser.rs:1254: replace <impl Serializer for DateStrEmitter<'a, 'b>>::serialize_str -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.938s
src/map.rs:214: replace <impl Clone for Map<String, Value>>::clone -> Self with Default::default() ... NOT CAUGHT in 1.842s
src/value.rs:1026: replace <impl SerializeStruct for SerializeMap>::serialize_field -> Result<(), crate::ser::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.805s
src/ser.rs:1123: replace <impl SerializeMap for SerializeTable<'a, 'b>>::end -> Result<(), Error> with Ok(Default::default()) ... NOT CAUGHT in 1.858s
src/ser.rs:804: replace <impl Serializer for &'b mut Serializer<'a>>::serialize_i64 -> Result<(), Self::Error> with Ok(Default::default()) ... NOT CAUGHT in 1.841s
src/datetime.rs:193: replace <impl Display for Datetime>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 2.108s
src/ser.rs:177: replace StringSettings::pretty -> StringSettings with Default::default() ... NOT CAUGHT in 2.018s
481 mutants tested in 9:43: 197 missed, 58 caught, 225 unviable, 1 timeouts

Conclusions:

itoa

https://github.com/dtolnay/itoa

2022-08-06, itoa 1.0.3

Again seems like it should be testable and well-tested; let's try.

Freshen source tree ... ok in 0.008s
Copy source and build products to scratch directory ... 15 MB in 0.011s
Unmutated baseline ... ok in 0.159s
Auto-set test timeout to 20.0s
Found 4 mutants to test
src/lib.rs:70: replace <impl Clone for Buffer>::clone -> Self with Default::default() ... NOT CAUGHT in 0.454s
src/lib.rs:86: replace Buffer::format -> &str with Default::default() ... caught in 0.301s
src/udiv128.rs:3: replace u128_mulhi -> u128 with Default::default() ... caught in 0.299s
src/udiv128.rs:29: replace udivmod_1e19 -> (u128, u64) with Default::default() ... caught in 0.267s
4 mutants tested in 0:01: 1 missed, 3 caught

It's surprising that it finds so few mutants! The reason is that a lot of the code is generated by macros and cargo-mutants, probably reasonably, doesn't know how to mutate them. (Conceivably we could try to mutate the MIR...)

The mutant that's missed, impl Clone for Buffer, is at least a little bit interesting: the code always returns a new buffer, which is the same as what the mutation does. Normally, it would seem like a bug if Clone doesn't copy state from the existing object. However in this case the buffer basically has no state in its public behavior, so it's not a bug.

semver

https://github.com/dtolnay/semver

2022-08-06, semver 1.0.13

Completes quickly and finds some potentially interesting mutants.

Freshen source tree ... ok in 1.020s
Copy source and build products to scratch directory ... 60 MB in 0.039s
Unmutated baseline ... ok in 0.961s
Auto-set test timeout to 20.0s
Found 105 mutants to test
src/eval.rs:26: replace matches_comparator -> bool with true ... NOT CAUGHT in 0.686s
src/serde.rs:69: replace <impl Deserialize for VersionReq>::deserialize::<impl Visitor for VersionReqVisitor>::expecting -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 0.768s
src/serde.rs:43: replace <impl Deserialize for Version>::deserialize::<impl Visitor for VersionVisitor>::expecting -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 0.653s
src/eval.rs:26: replace matches_comparator -> bool with false ... NOT CAUGHT in 0.684s
src/display.rs:117: replace <impl Debug for BuildMetadata>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 0.713s
src/serde.rs:76: replace <impl Deserialize for VersionReq>::deserialize::<impl Visitor for VersionReqVisitor>::visit_str -> Result<Self::Value, E> with Ok(Default::default()) ... NOT CAUGHT in 0.704s
src/serde.rs:37: replace <impl Deserialize for Version>::deserialize -> Result<Self, D::Error> with Ok(Default::default()) ... NOT CAUGHT in 0.703s
src/serde.rs:19: replace <impl Serialize for VersionReq>::serialize -> Result<S::Ok, S::Error> with Ok(Default::default()) ... NOT CAUGHT in 0.704s
src/impls.rs:34: replace <impl Deref for BuildMetadata>::deref -> &Self::Target with Default::default() ... NOT CAUGHT in 0.684s
src/display.rs:94: replace <impl Debug for Version>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 0.776s
src/backport.rs:8: replace <impl StripPrefixExt for str>::strip_prefix -> Option<&str> with Default::default() ... NOT CAUGHT in 0.754s
src/impls.rs:152: replace <impl FromIterator for VersionReq>::from_iter -> Self with Default::default() ... NOT CAUGHT in 0.704s
src/serde.rs:95: replace <impl Deserialize for Comparator>::deserialize::<impl Visitor for ComparatorVisitor>::expecting -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 0.735s
src/serde.rs:28: replace <impl Serialize for Comparator>::serialize -> Result<S::Ok, S::Error> with Ok(Default::default()) ... NOT CAUGHT in 0.686s
src/error.rs:103: replace <impl Debug for Error>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 0.736s
src/serde.rs:89: replace <impl Deserialize for Comparator>::deserialize -> Result<Self, D::Error> with Ok(Default::default()) ... NOT CAUGHT in 0.723s
src/backport.rs:37: replace alloc::alloc::alloc -> * mut u8 with Default::default() ... NOT CAUGHT in 0.670s
src/identifier.rs:219: replace <impl Drop for Identifier>::drop with () ... NOT CAUGHT in 0.704s
src/lib.rs:494: replace Comparator::matches -> bool with false ... NOT CAUGHT in 0.718s
src/serde.rs:50: replace <impl Deserialize for Version>::deserialize::<impl Visitor for VersionVisitor>::visit_str -> Result<Self::Value, E> with Ok(Default::default()) ... NOT CAUGHT in 0.685s
src/backport.rs:46: replace alloc::alloc::dealloc with () ... NOT CAUGHT in 0.693s
src/display.rs:111: replace <impl Debug for Prerelease>::fmt -> fmt::Result with Ok(Default::default()) ... NOT CAUGHT in 0.738s
src/impls.rs:18: replace <impl Hash for Identifier>::hash with () ... NOT CAUGHT in 0.704s
src/backport.rs:31: replace alloc::alloc::Layout::from_size_align_unchecked -> Self with Default::default() ... NOT CAUGHT in 0.736s
src/serde.rs:102: replace <impl Deserialize for Comparator>::deserialize::<impl Visitor for ComparatorVisitor>::visit_str -> Result<Self::Value, E> with Ok(Default::default()) ... NOT CAUGHT in 0.718s
src/serde.rs:10: replace <impl Serialize for Version>::serialize -> Result<S::Ok, S::Error> with Ok(Default::default()) ... NOT CAUGHT in 0.708s
src/lib.rs:494: replace Comparator::matches -> bool with true ... NOT CAUGHT in 0.685s
src/serde.rs:63: replace <impl Deserialize for VersionReq>::deserialize -> Result<Self, D::Error> with Ok(Default::default()) ... NOT CAUGHT in 0.685s
105 mutants tested in 0:57: 28 missed, 62 caught, 15 unviable

For example Comparator::matches -> bool with true seems like a pretty important thing to test? impl Hash being wrong could matter, if you care that these things are hashed. Maybe serialization is not tested enough.

Conclusions:

  • Follow up; see how many could be tested; maybe offer the findings.
  • Perhaps there should be an option to skip impl Debug. You might want to test it, but perhaps for some crates it's not really a priority.

syn

https://github.com/dtolnay/syn

2022-08-06, syn 1.0.99

Another amazingly useful dtolnay@ crate.

I suspect this will be slower to test and find some gaps but it will succeed...

Actually this fails to compile but with a really nice message:

   Compiling syn-test-suite v0.0.0 (/home/mbp/src/syn/tests/features)

NOTE:  use --release
  Syn's test suite has some tests that run on every source file
  and test case in the rust-lang/rust repo, which can be pretty
  slow in debug mode. Consider running cargo test with `--release`
  to speed things up.

ERROR:  use --all-features
  Syn's test suite normally only works with all-features enabled.
  Run again with `--all-features`, or run with `--features test`
  to bypass this check.

So we need to pass those options.

2022-08-07: With the recent addition of --cargo-arg we can pass those two options and then it does build and start testing mutants. It finds 2327 mutants, and they take a while to test, so it will take a while. A fair fraction seem uncaught.

papermario-solver

https://github.com/kupiakos/papermario-solver

cargo-mutants 0.2.11, 2022-08-20

This crate is an example of a Rust/TypeScript WASM web app.

On the first attempt cargo-mutants generated no mutants, and --list-files showed that it didn't even see any source files.

This turned out to be because this crate, as seems to be typical for WASM crates, has a cdylib not a lib or bin target, and cargo-mutants didn't previously consider them candidates for mutation. I fixed that in https://github.com/sourcefrog/cargo-mutants/issues/78.

With this fixed, it finds that no mutants are caught, which is honestly fine because this is a toy or game. But, it's interesting that this code does have what look like doctests, in https://github.com/kupiakos/papermario-solver/blob/e9667dcfe7e595cfc5687d6f59ae2805dfd80819/src/solver.rs#L114-L125.

It turns out that tests aren't run in cdylib either, but that can be fixed by marking the target as both a cdylib and a lib:

diff --git a/Cargo.toml b/Cargo.toml
index 47c71d0..9020bfe 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -9,7 +9,7 @@ readme = "README.md"
 edition = "2018"

 [lib]
-crate-type = ["cdylib"]
+crate-type = ["cdylib", "lib"]
 path = "src/solver.rs"

With that inserted, it turns out the doctest fails when run, because it's calling private functions, and doctests are linked into separate binaries that can only access the library's public API. Again, this is fine because the code fragment in the docstring is really meant more as documentation than as a test. But it is nice that cargo-mutants helped highlight that the apparent test was silently "broken" (to the extent that it looks like a test.)

Clone this wiki locally