Skip to content

Commit

Permalink
concretizations -> generic-args, impl-concretizations -> `impl-…
Browse files Browse the repository at this point in the history
…generic-args`
  • Loading branch information
smoelius committed Jan 5, 2024
1 parent dafbee1 commit 36f6c46
Show file tree
Hide file tree
Showing 13 changed files with 182 additions and 147 deletions.
40 changes: 18 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,27 +86,27 @@ The primary effects of the `test_fuzz` macro are:

Impose `where_predicates` (e.g., trait bounds) on the struct used to serialize/deserialize arguments. This may be necessary, e.g., if a target's argument type is an associated type. For an example, see [associated_type.rs] in this repository.

##### `concretize = "parameters"`
##### `generic_args = "parameters"`

Use `parameters` as the target's type parameters when fuzzing. Example:

```rust
#[test_fuzz(concretize = "String")]
#[test_fuzz(generic_args = "String")]
fn foo<T: Clone + Debug + Serialize>(x: &T) {
...
}
```

Note: The target's arguments must be serializable for **every** instantiation of its type parameters. But the target's arguments are required to be deserializable only when the target is instantiated with `parameters`.

##### `concretize_impl = "parameters"`
##### `impl_generic_args = "parameters"`

Use `parameters` as the target's `Self` type parameters when fuzzing. Example:

```rust
#[test_fuzz_impl]
impl<T: Clone + Debug + Serialize> for Foo {
#[test_fuzz(concretize_impl = "String")]
#[test_fuzz(impl_generic_args = "String")]
fn bar(&self, x: &T) {
...
}
Expand Down Expand Up @@ -170,13 +170,13 @@ Calling the target in this way allows `function` to set up the call's environmen

Do not try to [auto-generate corpus files] for the target.

##### `only_concretizations`
##### `only_generic_args`

Record the target's concretizations when running tests, but do not generate corpus files and do not implement a fuzzing harness. This can be useful when the target is a generic function, but it is unclear what type parameters should be used for fuzzing.
Record the target's generic args when running tests, but do not generate corpus files and do not implement a fuzzing harness. This can be useful when the target is a generic function, but it is unclear what type parameters should be used for fuzzing.

The intended workflow is: enable `only_concretizations`, then run `cargo test` followed by `cargo test-fuzz --display concretizations`. One of the resulting concretizations might be usable as `concretize`'s `parameters`. Similarly, a concretization resulting from `cargo test-fuzz --display impl-concretizations` might be usable as `concretize_impl`'s `parameters`.
The intended workflow is: enable `only_generic_args`, then run `cargo test` followed by `cargo test-fuzz --display generic-args`. One of the resulting generic args might be usable as `generic_args`'s `parameters`. Similarly, generic args resulting from `cargo test-fuzz --display impl-generic-args` might be usable as `impl_generic_args`'s `parameters`.

Note, however, that just because a target was concretized with certain parameters during tests, it does not imply the target's arguments are serializable/deserializable when so concretized. The results of `--display concretizations`/`--display impl-concretizations` are merely suggestive.
Note, however, that just because a target was called with certain parameters during tests, it does not imply the target's arguments are serializable/deserializable when those parameters are used. The results of `--display generic-args`/`--display impl-generic-args` are merely suggestive.

##### `rename = "name"`

Expand Down Expand Up @@ -255,12 +255,10 @@ Options:
--backtrace Display backtraces
--consolidate Move one target's crashes, hangs, and work queue to its corpus; to
consolidate all targets, use --consolidate-all
--display <OBJECT> Display concretizations, corpus, crashes, `impl` concretizations,
hangs, or work queue. By default, corpus uses an uninstrumented
fuzz target; the others use an instrumented fuzz target. To
display the corpus with instrumentation, use --display
corpus-instrumented. [possible values: concretizations, corpus,
corpus-instrumented, crashes, hangs, impl-concretizations, queue]
--display <OBJECT> Display corpus, crashes, generic args, `impl` generic args, hangs,
or work queue. By default, corpus uses an uninstrumented fuzz
target; the others use an instrumented fuzz target. To display the
corpus with instrumentation, use --display corpus-instrumented.
--exact Target name is an exact name rather than a substring
--exit-code Exit with 0 if the time limit was reached, 1 for other
programmatic aborts, and 2 if an error occurred; implies --no-ui,
Expand All @@ -279,9 +277,7 @@ Options:
--replay <OBJECT> Replay corpus, crashes, hangs, or work queue. By default, corpus
uses an uninstrumented fuzz target; the others use an instrumented
fuzz target. To replay the corpus with instrumentation, use
--replay corpus-instrumented. [possible values: concretizations,
corpus, corpus-instrumented, crashes, hangs, impl-concretizations,
queue]
--replay corpus-instrumented.
--reset Clear fuzzing data for one target, but leave corpus intact; to
reset all targets, use --reset-all
--resume Resume target's last fuzzing session
Expand Down Expand Up @@ -467,15 +463,15 @@ A target's arguments must implement the [`Clone`] trait. The reason for this req

### Serializable / deserializable arguments

In general, a target's arguments must implement the [`serde::Serialize`] and [`serde::Deserialize`] traits, e.g., by [deriving them]. We say "in general" because `test-fuzz` knows how to handle certain special cases that wouldn't normally be serializable/deserializable. For example, an argument of type `&str` is converted to `String` when serializing, and back to a `&str` when deserializing. See also [`concretize`] and [`concretize_impl`] above.
In general, a target's arguments must implement the [`serde::Serialize`] and [`serde::Deserialize`] traits, e.g., by [deriving them]. We say "in general" because `test-fuzz` knows how to handle certain special cases that wouldn't normally be serializable/deserializable. For example, an argument of type `&str` is converted to `String` when serializing, and back to a `&str` when deserializing. See also [`generic_args`] and [`impl_generic_args`] above.

### Global variables

The fuzzing harnesses that `test-fuzz` implements do not initialize global variables. While [`execute_with`] provides some remedy, it is not a complete solution. In general, fuzzing a function that relies on global variables requires ad-hoc methods.

### [`convert`] and [`concretize`] / [`concretize_impl`]
### [`convert`] and [`generic_args`] / [`impl_generic_args`]

These options are incompatible in the following sense. If a fuzz target's argument type is a type parameter, [`convert`] will try to match the type parameter, not the type to which it is concretized. Supporting the latter would seem to require simulating type substitution as the compiler would perform it. However, this is not currently implemented.
These options are incompatible in the following sense. If a fuzz target's argument type is a type parameter, [`convert`] will try to match the type parameter, not the type to which the parameter is set. Supporting the latter would seem to require simulating type substitution as the compiler would perform it. However, this is not currently implemented.

## Tips and tricks

Expand Down Expand Up @@ -516,15 +512,15 @@ These options are incompatible in the following sense. If a fuzz target's argume
[`cargo test-fuzz` command]: #cargo-test-fuzz-command
[`cargo test-fuzz`]: #cargo-test-fuzz-command
[`cargo-clone`]: https://github.com/JanLikar/cargo-clone
[`concretize_impl`]: #concretize_impl--parameters
[`concretize`]: #concretize--parameters
[`convert`]: #convert--x-y
[`core::ops::Add`]: https://doc.rust-lang.org/beta/core/ops/trait.Add.html
[`core::ops::Div`]: https://doc.rust-lang.org/beta/core/ops/trait.Div.html
[`core::ops::Sub`]: https://doc.rust-lang.org/beta/core/ops/trait.Sub.html
[`deserialize_with`]: https://serde.rs/field-attrs.html#deserialize_with
[`enable_in_production`]: #enable_in_production
[`execute_with`]: #execute_with--function
[`generic_args`]: #generic_args--parameters
[`impl_generic_args`]: #impl_generic_args--parameters
[`num_traits::One`]: https://docs.rs/num-traits/0.2.14/num_traits/identities/trait.One.html
[`num_traits::bounds::Bounded`]: https://docs.rs/num-traits/0.2.14/num_traits/bounds/trait.Bounded.html
[`proc_macro_span`]: https://github.com/rust-lang/rust/issues/54725
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ index b4a5d9e..1c421e1 100644
@@ -176,8 +181,12 @@ impl<Hash: hash::Hash + Member + Serialize, Ex> ReadyTransactions<Hash, Ex> {
/// that are in this queue.
/// Returns transactions that were replaced by the one imported.
+ #[test_fuzz::test_fuzz(concretize_impl = "u64, Vec<u8>", bounds = "Hash: Eq + hash::Hash")]
+ #[test_fuzz::test_fuzz(impl_generic_args = "u64, Vec<u8>", bounds = "Hash: Eq + hash::Hash")]
pub fn import(
&mut self,
tx: WaitingTransaction<Hash, Ex>,
Expand Down
2 changes: 1 addition & 1 deletion cargo-test-fuzz/patches/substrate_node_template.patch
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ index 9550d3d..4306d04 100644
#[pallet::weight(T::WeightInfo::do_something())]
+ #[test_fuzz::test_fuzz(
+ bounds = "T: frame_system::Config",
+ concretize_impl = "crate::mock::Test",
+ impl_generic_args = "crate::mock::Test",
+ convert = "OriginFor<T>, SerializableAccountId<AccountIdFor<T>>",
+ execute_with = "crate::mock::new_test_ext().execute_with"
+ )]
Expand Down
36 changes: 17 additions & 19 deletions cargo-test-fuzz/src/bin/cargo_test_fuzz/transition.rs
Original file line number Diff line number Diff line change
Expand Up @@ -37,8 +37,9 @@ struct TestFuzzWithDeprecations {
#[arg(
long,
value_name = "OBJECT",
help = "Display concretizations, corpus, crashes, `impl` concretizations, hangs, or work \
queue. By default, corpus uses an uninstrumented fuzz target; the others use an \
hide_possible_values = true,
help = "Display corpus, crashes, generic args, `impl` generic args, hangs, or work queue. \
By default, corpus uses an uninstrumented fuzz target; the others use an \
instrumented fuzz target. To display the corpus with instrumentation, use \
--display corpus-instrumented."
)]
Expand Down Expand Up @@ -88,6 +89,7 @@ struct TestFuzzWithDeprecations {
#[arg(
long,
value_name = "OBJECT",
hide_possible_values = true,
help = "Replay corpus, crashes, hangs, or work queue. By default, corpus uses an \
uninstrumented fuzz target; the others use an instrumented fuzz target. To replay \
the corpus with instrumentation, use --replay corpus-instrumented."
Expand Down Expand Up @@ -191,31 +193,27 @@ impl From<TestFuzzWithDeprecations> for super::TestFuzz {
}
}

#[allow(unused_macros)]
macro_rules! process_deprecated_action_object {
($opts:ident, $action:ident, $object:ident) => {
paste::paste! {
if $opts.[< $action _ $object >] {
use heck::ToKebabCase;
eprintln!(
"`--{}-{}` is deprecated. Use `--{} {}` (no hyphen).",
stringify!($action),
stringify!($object).to_kebab_case(),
stringify!($action),
stringify!($object).to_kebab_case(),
);
if $opts.$action.is_none() {
$opts.$action = Some(Object::[< $object:camel >]);
}
}
($opts:ident, $action:ident, $object_old:ident, $object_new:ident) => {
if $opts.$action == Some(Object::$object_old) {
use heck::ToKebabCase;
eprintln!(
"{}` is deprecated. Use `{}`.",
stringify!($object_old).to_kebab_case(),
stringify!($object_new).to_kebab_case(),
);
$opts.$action = Some(Object::$object_new);
}
};
}

#[allow(deprecated)]
pub(crate) fn cargo_test_fuzz<T: AsRef<OsStr>>(args: &[T]) -> Result<()> {
#[allow(unused_mut)]
let SubCommand::TestFuzz(mut opts) = Opts::parse_from(args).subcmd;

process_deprecated_action_object!(opts, display, Concretizations, GenericArgs);
process_deprecated_action_object!(opts, display, ImplConcretizations, ImplGenericArgs);

super::run(super::TestFuzz::from(opts))
}

Expand Down
31 changes: 18 additions & 13 deletions cargo-test-fuzz/src/lib.rs
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#![cfg_attr(dylint_lib = "general", allow(crate_wide_allow))]
#![allow(clippy::use_self)]
#![allow(deprecated)]
#![deny(clippy::expect_used)]
#![deny(clippy::unwrap_used)]
#![warn(clippy::panic)]
Expand All @@ -12,9 +12,9 @@ use cargo_metadata::{
use clap::{crate_version, ValueEnum};
use heck::ToKebabCase;
use internal::dirs::{
concretizations_directory_from_target, corpus_directory_from_target,
crashes_directory_from_target, hangs_directory_from_target,
impl_concretizations_directory_from_target, output_directory_from_target,
corpus_directory_from_target, crashes_directory_from_target,
generic_args_directory_from_target, hangs_directory_from_target,
impl_generic_args_directory_from_target, output_directory_from_target,
queue_directory_from_target, target_directory,
};
use log::debug;
Expand Down Expand Up @@ -55,12 +55,16 @@ bitflags! {
#[derive(Clone, Copy, Debug, Display, Deserialize, PartialEq, Eq, Serialize, ValueEnum)]
#[remain::sorted]
pub enum Object {
#[deprecated]
Concretizations,
Corpus,
CorpusInstrumented,
Crashes,
GenericArgs,
Hangs,
#[deprecated]
ImplConcretizations,
ImplGenericArgs,
Queue,
}

Expand Down Expand Up @@ -136,7 +140,7 @@ pub fn run(opts: TestFuzz) -> Result<()> {
if opts.list
|| matches!(
opts.display,
Some(Object::Corpus | Object::ImplConcretizations | Object::Concretizations)
Some(Object::Corpus | Object::ImplGenericArgs | Object::GenericArgs)
)
|| opts.replay == Some(Object::Corpus)
{
Expand All @@ -147,10 +151,7 @@ pub fn run(opts: TestFuzz) -> Result<()> {

if let Some(object) = opts.replay {
ensure!(
!matches!(
object,
Object::ImplConcretizations | Object::Concretizations
),
!matches!(object, Object::ImplGenericArgs | Object::GenericArgs),
"`--replay {}` is invalid.",
object.to_string().to_kebab_case()
);
Expand Down Expand Up @@ -697,22 +698,26 @@ fn reset(opts: &TestFuzz, executable_targets: &[(Executable, Vec<String>)]) -> R
Ok(())
}

#[allow(clippy::panic)]
fn flags_and_dir(object: Object, krate: &str, target: &str) -> (Flags, PathBuf) {
match object {
Object::Concretizations | Object::ImplConcretizations => {
panic!("`{object}` should have been filter out")
}
Object::Corpus | Object::CorpusInstrumented => (
Flags::REQUIRES_CARGO_TEST,
corpus_directory_from_target(krate, target),
),
Object::Crashes => (Flags::empty(), crashes_directory_from_target(krate, target)),
Object::Hangs => (Flags::empty(), hangs_directory_from_target(krate, target)),
Object::Queue => (Flags::empty(), queue_directory_from_target(krate, target)),
Object::ImplConcretizations => (
Object::ImplGenericArgs => (
Flags::REQUIRES_CARGO_TEST | Flags::RAW,
impl_concretizations_directory_from_target(krate, target),
impl_generic_args_directory_from_target(krate, target),
),
Object::Concretizations => (
Object::GenericArgs => (
Flags::REQUIRES_CARGO_TEST | Flags::RAW,
concretizations_directory_from_target(krate, target),
generic_args_directory_from_target(krate, target),
),
}
}
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,4 @@
use internal::dirs::{
concretizations_directory_from_target, impl_concretizations_directory_from_target,
};
use internal::dirs::{generic_args_directory_from_target, impl_generic_args_directory_from_target};
use std::fs::remove_dir_all;
use testing::{examples, CommandExt};

Expand All @@ -25,8 +23,8 @@ fn generic() {
);
test(
"generic",
"test_only_concretizations",
"target_only_concretizations",
"test_only_generic_args",
"target_only_generic_args",
&impl_expected,
&expected,
);
Expand All @@ -48,25 +46,25 @@ fn unserde() {
}

fn test(krate: &str, test: &str, target: &str, impl_expected: &[&str], expected: &[&str]) {
let impl_concretizations = impl_concretizations_directory_from_target(krate, target);
let impl_generic_args = impl_generic_args_directory_from_target(krate, target);

// smoelius: `corpus` is distinct for all tests. So there is no race here.
#[cfg_attr(dylint_lib = "general", allow(non_thread_safe_call_in_test))]
remove_dir_all(impl_concretizations).unwrap_or_default();
remove_dir_all(impl_generic_args).unwrap_or_default();

let concretizations = concretizations_directory_from_target(krate, target);
let generic_args = generic_args_directory_from_target(krate, target);

#[cfg_attr(dylint_lib = "general", allow(non_thread_safe_call_in_test))]
remove_dir_all(concretizations).unwrap_or_default();
remove_dir_all(generic_args).unwrap_or_default();

examples::test(krate, test)
.unwrap()
.logged_assert()
.success();

for (option, expected) in &[
("--display=impl-concretizations", impl_expected),
("--display=concretizations", expected),
("--display=impl-generic-args", impl_expected),
("--display=generic-args", expected),
] {
let assert = &examples::test_fuzz(krate, target)
.unwrap()
Expand Down
2 changes: 1 addition & 1 deletion examples/tests/associated_type.rs
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ where
}
}

#[test_fuzz::test_fuzz(concretize = "Address", bounds = "T: Serializable")]
#[test_fuzz::test_fuzz(generic_args = "Address", bounds = "T: Serializable")]
fn serializes_to<T>(x: &T, y: &T::Out) -> bool
where
T: Clone + DeserializeOwned + Serialize + Serializable,
Expand Down

0 comments on commit 36f6c46

Please sign in to comment.