clarinet fuzz command and heterogeneous test-suites

[moodmosaic]

This was started as two separate GitHub issues, but then I thought I should merge them. — The idea is:

clarinet test can support property tests, and those tests may be written in either TypeScript or Clarity
clarinet fuzz can then turn those tests into fuzz tests

Context

Clarinet tests are essentially Deno tests. This has the interesting side-effect of being able to use what's already available in JS/TS ecosystem when testing Clarity code. (For example, fast-check and dspec can be used, as we've done here with @LNow.)

A typical workflow is declaring functions in Clarity and then writing tests in TypeScript. Clarinet can then discover those tests (with the help of Deno) and run them. It would be practical however to also discover tests written in Clarity and run those as well.

Discoverability

In addition to existing test suite(s) in TypeScript, Clarinet can execute as a test any Clarity function that meets the following criteria:

• The method is public
• The method name starts with test (configurable in Clarinet.toml)

(If a public function named beforeEach is present it can be executed before each test is run. Such a function can exist in the style of testing @jcnelson does here, may be discussed also on a separate thread, however.)

type	prefix	arguments	semantics
concrete	test	no	single execution with concrete values
property	test	yes	multiple executions with randomly generated concrete values (smaller values)
fuzz	test	yes	multiple executions with randomly generated concrete values (biased)*

^*Also configurable, in Clarinet.toml.

Fuzz mode can be enabled via new clarinet fuzz command. Essentially it can work the same as clarinet test but use a different fast-check configuration.

Libraries

Both test suites (TypeScript, and Clarity (hypothetically)) can be discovered and run from the context of Deno, and in the case of property tests and fuzz tests, the generated data can be provided by fast-check.

I have been using (and getting an inside look into) fast-check recently. It has all the modern features of a prop/fuzz testing library, e.g. model testing, integrated shrinking, control over the scope of generated values, and many other useful functions.

Pros and cons

What are the advantages and disadvantages of having the option to write tests in Clarity?

• One disadvantage that @LNow pointed out to me is that this can screw the execution costs.
• An advantage can be that there's less context switching; you write your contract in Clarity and your tests in Clarity as well. (No need to marshal stuff between different languages (as done here for example).
• The possibility to encode (some of the) smart contract audits in code that (continuously) runs and checks those tests, instead of having all the audits based on a specific git commit and PDF file.
• (more pros and cons to be added)...

[moodmosaic]

Observing Test Case Distribution

It is important to be aware of the distribution of test cases: if the test data is not well distributed then conclusions drawn from the test results may be invalid.

Based on my current research, fast-check can monitor the underlying test data but that's separate from the actual test run(s); it is purely informational and doesn’t have a threshold below which it will fail the test(s).

"Hello, world!" example

Monitor how frequently each Clarinet account gets picked up by the generator below. According to the docs, this generator makes each account be almost equally likely chosen:

Clarinet.test({
  name: "fc.statistics/fc.constantFrom/clarinet.accounts",
  async fn(_: Chain, accounts: Map<string, Account>) {
    fc.statistics(
      fc.constantFrom(...accounts.values()),
      (account) =>
          account.name === "deployer" ? "deployer"
        : account.name === "wallet_1" ? "wallet_1"
        : account.name === "wallet_2" ? "wallet_2"
        : account.name === "wallet_3" ? "wallet_3"
        : account.name === "wallet_4" ? "wallet_4"
        : account.name === "wallet_5" ? "wallet_5"
        : account.name === "wallet_6" ? "wallet_6"
        : account.name === "wallet_7" ? "wallet_7"
        : account.name === "wallet_8" ? "wallet_8"
        : account.name === "wallet_9" ? "wallet_9"
        : "wallet_10",
      { numRuns: 1000, unbiased: true },
    );
  },
});

Prints:

$ clarinet test
Running ../test.ts
deployer..10.17%
wallet_8..10.13%
wallet_6..10.05%
wallet_5..10.05%
wallet_4..10.00%
wallet_1...9.95%
wallet_2...9.95%
wallet_9...9.92%
wallet_3...9.91%
wallet_7...9.88%
* fc.statistics/fc.constantFrom/clarinet.accounts ... ok (189ms)

The first step towards

The first step towards supporting property-based (and fuzz) tests is getting something like this to compiletype-check (and run!):

@@ -156,14 +156,12 @@ function CargoCommands(accounts: Map<string, Account>) {
 //   },
 // });
 
 Clarinet.test({
   name: "fc.statistics/fc.constantFrom/clarinet.accounts",
-  async fn(_: Chain, accounts: Map<string, Account>) {
+  async fn(_: Chain, accounts: Map<string, Account>, account: Account) {
     fc.statistics(
-      fc.constantFrom(...accounts.values()),
-      (account) =>
           account.name === "deployer" ? "deployer"
         : account.name === "wallet_1" ? "wallet_1"
         : account.name === "wallet_2" ? "wallet_2"
         : account.name === "wallet_3" ? "wallet_3"
         : account.name === "wallet_4" ? "wallet_4"

And perhaps now it's time for a checklist. (To be added in the next comment.)

[moodmosaic]

In order to separate concrete (current) tests from property/fuzz-based ones, the default signature of the test method should change so that there's no parameters in concrete tests:

@@ -2,21 +2,21 @@
 import { Clarinet, Tx, Chain, Account, types } from 'https://deno.land/x/clarinet@v0.31.0/index.ts';
 import { assertEquals } from 'https://deno.land/std@0.90.0/testing/asserts.ts';
 
 Clarinet.test({
     name: "Ensure that <...>",
-    async fn(chain: Chain, accounts: Map<string, Account>) {
-        let block = chain.mineBlock([
+    async fn() {
+        let block = ctx.chain.mineBlock([
             /* 
              * Add transactions with: 
              * Tx.contractCall(...)
             */
         ]);
         assertEquals(block.receipts.length, 0);
         assertEquals(block.height, 2);
 
-        block = chain.mineBlock([
+        block = ctx.chain.mineBlock([
             /* 
              * Add transactions with: 
              * Tx.contractCall(...)
             */
         ]);

Then chain, accounts, and contracts, can be referenced from a Context object or equivalent. See our work with @LNow here for an example.

[moodmosaic]

Using the latest from #511 by @lgalabru, I took a stub at implementing a Clarinet.fuzz method, assuming that reflection in TypeScript/Deno could work quite similar to what we've used to (CLR, JVM).

Step 1: Take an existing Clarinet test

Clarinet.test({
  name: 'write-sup returns expected string',
  async fn(chain: Chain, accounts: Map<string, Account>) {
    // Arrange
    const account = accounts.get('deployer')!;
    const msg = types.utf8("lorem ipsum");
    const stx = types.uint(123);

    // Act
    const block = chain.mineBlock([
      Tx.contractCall(
        'sup', 'write-sup', [msg, stx], account.address)
    ]);
    const result = block.receipts[0].result;

    // Assert
    result
      .expectOk()
      .expectAscii('Sup written successfully');
  }
});

Step 2: Change test to fuzz (or prop, to be discussed)

-Clarinet.test({
+Clarinet.fuzz({
   name: 'write-sup returns expected string',
   async fn(chain: Chain, accounts: Map<string, Account>) {
     // Arrange
     const account = accounts.get('deployer')!;
     const msg = types.utf8("lorem ipsum");

Step 3: Specify number of runs (optional - default is 100)

 Clarinet.fuzz({
   name: 'write-sup returns expected string',
+  runs: 10,
   async fn(chain: Chain, accounts: Map<string, Account>) {
     // Arrange
     const account = accounts.get('deployer')!;
     const msg = types.utf8("lorem ipsum");
     const stx = types.uint(123);

Step 4: Auto-generate values instead of hardcoding them

 Clarinet.fuzz({
   name: 'write-sup returns expected string',
   runs: 10,
-  async fn(chain: Chain, accounts: Map<string, Account>) {
+  async fn(chain: Chain, account: Account, message: string, howMuch: number|bigint) {
     // Arrange
-    const account = accounts.get('deployer')!;
-    const msg = types.utf8("lorem ipsum");
-    const stx = types.uint(123);
+    const msg = types.utf8(message);
+    const stx = types.uint(howMuch);
 
     // Act
     const block = chain.mineBlock([
       Tx.contractCall(
         'sup', 'write-sup', [msg, stx], account.address)

---

The test should now look like below:

Clarinet.fuzz({
  name: 'write-sup returns expected string',
  runs: 10,
  async fn(chain: Chain, account: Account, message: string, howMuch: number|bigint) {
    // Arrange
    const msg = types.utf8(message);
    const stx = types.uint(howMuch);

    // Act
    const block = chain.mineBlock([
      Tx.contractCall(
        'sup', 'write-sup', [msg, stx], account.address)
    ]);
    const result = block.receipts[0].result;

    // Assert
    result
      .expectOk()
      .expectAscii('Sup written successfully');
  }
});

Output:

$ clarinet test
./tests/sup_test.ts => write-sup returns expected string ... #1 wallet_9 12 ipsum semper ultricies ante proin arcu congue ut maecenas est maecenas placerat ... ok (17ms)
./tests/sup_test.ts => write-sup returns expected string ... #2 wallet_3 98 ultricies dolor pharetra a cras placerat ... ok (13ms)
./tests/sup_test.ts => write-sup returns expected string ... #3 wallet_1 97 tempor erat ... ok (21ms)
./tests/sup_test.ts => write-sup returns expected string ... #4 wallet_3 99 ipsum consectetuer orci ... ok (15ms)
./tests/sup_test.ts => write-sup returns expected string ... #5 wallet_8 27 molestie eros ... ok (15ms)
./tests/sup_test.ts => write-sup returns expected string ... #6 wallet_5 13 risus enim aliquam aliquam fermentum varius arcu augue vivamus varius ... ok (15ms)
./tests/sup_test.ts => write-sup returns expected string ... #7 wallet_8 16 vivamus ut ... ok (10ms)
./tests/sup_test.ts => write-sup returns expected string ... #8 deployer 46 suscipit consectetur non et orci ... ok (8ms)
./tests/sup_test.ts => write-sup returns expected string ... #9 wallet_9 55 dolor ... ok (10ms)
./tests/sup_test.ts => write-sup returns expected string ... #10 wallet_6 88 tincidunt ... ok (12ms)
./tests/sup_test.ts => write-sup returns expected string ... ok (164ms)

ok | 1 passed (10 steps) | 0 failed (218ms)

---

Note that write-sup comes from @kenrogers article and is defined as:

(define-public (write-sup (message (string-utf8 500)) (price uint))
    (begin
        (
            try! (stx-transfer? price tx-sender receiver-address)
        )

        ;; #[allow(unchecked_data)]
        (map-set messages tx-sender message )

        (var-set total-sups (+ (var-get total-sups) u1))

        (ok "Sup written successfully")
    )
)

We can change runs to 200 hoping that a message longer than 500 chars will be sent from the fuzzer:

 Clarinet.fuzz({
   name: 'write-sup returns expected string',
-  runs: 10,
+  runs: 200,
   async fn(chain: Chain, account: Account, message: string, howMuch: number|bigint) {
     // Arrange
     const msg = types.utf8(message);
     const stx = types.uint(howMuch);

./tests/sup_test.ts => write-sup returns expected string ... #184 wallet_4 94 non luctus nonummy fermentum maecenas justo praesent mauris ut ut fermentum egestas ... ok (6ms)
./tests/sup_test.ts => write-sup returns expected string ... #185 wallet_2 12 nec faucibus pretium ... FAILED (9ms)
    error: Error: Expected ok, got (err u2)
        throw new Error(
              ^
        at consume (file:///C:/Snapshot/dev/pub/stacks/sup/backend/tests/index.ts:589:11)
        at String.expectOk (file:///C:/Snapshot/dev/pub/stacks/sup/backend/tests/index.ts:602:10)
        at Object.fn (file:///C:/Snapshot/dev/pub/stacks/sup/backend/tests/sup_test.ts:61:8)
        at file:///C:/Snapshot/dev/pub/stacks/sup/backend/tests/index.ts:371:29
        at testStepSanitizer (deno:clarinet-cli/js/40_testing.js:442:13)
        at asyncOpSanitizer (deno:clarinet-cli/js/40_testing.js:142:15)
        at resourceSanitizer (deno:clarinet-cli/js/40_testing.js:368:13)
        at exitSanitizer (deno:clarinet-cli/js/40_testing.js:425:15)
        at TestContext.step (deno:clarinet-cli/js/40_testing.js:1328:19)
        at file:///C:/Snapshot/dev/pub/stacks/sup/backend/tests/index.ts:370:21
./tests/sup_test.ts => write-sup returns expected string ... #186 deployer 50 nulla suscipit ornare ultrices quis eu aenean ... ok (14ms)
./tests/sup_test.ts => write-sup returns expected string ... #187 wallet_4 10 blandit eros egestas sodales tortor ante pellentesque in ... ok (15ms)
./tests/sup_test.ts => write-sup returns expected string ... #188 wallet_3 63 felis fermentum suspendisse tellus diam ... ok (16ms)
./tests/sup_test.ts => write-sup returns expected string ... #189 wallet_9 84 sed curabitur metus aliquam integer eu felis purus nulla nulla ... ok (14ms)
./tests/sup_test.ts => write-sup returns expected string ... #190 wallet_1 99 tristique justo vel dolor scelerisque ... ok (15ms)
./tests/sup_test.ts => write-sup returns expected string ... #191 wallet_6 77 fermentum aliquam ipsum in ... ok (12ms)
./tests/sup_test.ts => write-sup returns expected string ... #192 wallet_1 13 nunc vel proin turpis ... ok (8ms)
./tests/sup_test.ts => write-sup returns expected string ... #193 wallet_6 95 non consequat luctus rhoncus ... ok (8ms)
./tests/sup_test.ts => write-sup returns expected string ... #194 deployer 35 aenean quam dolor ante ... ok (15ms)
./tests/sup_test.ts => write-sup returns expected string ... #195 wallet_6 94 bibendum purus rutrum dui pellentesque libero ... ok (8ms)
./tests/sup_test.ts => write-sup returns expected string ... #196 wallet_7 30 porttitor fusce sapien felis amet euismod eleifend feugiat ... ok (22ms)
./tests/sup_test.ts => write-sup returns expected string ... #197 deployer 10 justo egestas fermentum sed id nulla diam iaculis ... ok (14ms)
./tests/sup_test.ts => write-sup returns expected string ... #198 wallet_8 55 augue lorem convallis ... ok (16ms)
./tests/sup_test.ts => write-sup returns expected string ... #199 wallet_3 83 fusce proin maecenas ut nisl augue egestas sed lectus ut ante ... ok (12ms)
./tests/sup_test.ts => write-sup returns expected string ... #200 wallet_5 42 felis sapien lectus ... ok (16ms)
./tests/sup_test.ts => write-sup returns expected string ... FAILED (2s)

 ERRORS

write-sup returns expected string => ./tests/index.ts:311:10
error: Error: 13 test steps failed.
    at runTest (deno:clarinet-cli/js/40_testing.js:835:11)
    at async Object.runTests (deno:clarinet-cli/js/40_testing.js:1084:22)

 FAILURES

write-sup returns expected string => ./tests/index.ts:311:10

FAILED | 0 passed (187 steps) | 1 failed (13 steps) (2s)

error:: Test failed

[moodmosaic]

Next steps

Once deno config in Clarinet test is enabled via #511 (comment),

• Check that TS reflection actually works (e.g. a test method with many args, a test method with complex objects, etcetera)
• Check that shrinking works as expected (perhaps Deno steps are messing things up?)
• Configure/override fuzzer's auto-data generation (custom arbitraries) (@moodmosaic)
• Plug tst-reflect-transformer into deno (@Hookyns)
• Make sure each test runs autonomously (@lgalabru, @moodmosaic)

  Clarinet.fuzz({
    name: 'write-sup returns expected string',
    runs: 10, // Each run should (...or shouldn't?) reset the chain state?
    async fn(chain: Chain, accounts: Map<string, Account>, ...) { ... }

[moodmosaic]

Absolutely feel free to re-open and/or ping me in case you want help with this essential feature.

[lgalabru]

We are in the process of re-architecting clarinet testing approach, and our timeline is aggressive.
It feels like there could be an opportunity to support fuzzing from the get go, @moodmosaic if you're still interested, do you think you could sync with @hugocaillard?

[lgalabru]

More context here: #1022

[moodmosaic]

That's great news 👍

Yes, I could show to @hugocaillard all the pieces I got.

However, kindly note that I am currently on vacation and I may be slow to respond.

[hugocaillard]

Great idea!
@moodmosaic We could connect when you get back.
I only did some exploratory work so far and don't have a PR, but the idea is to bundle clarinet tesst features as wasm library, instead of embedding a JS runner (deno) in clarinet.

The current POC currently looks like that using Node 20 test runner (would also work we Jest, Mocha, etc)

import { main } from "../../../../hiro/clarinet/components/clarinet-sdk/dist/index.js";
// soon it will be `import { main } from "@clarinet-sdk/unit-test"` or smth like that
import { before, describe, it } from "node:test";
import assert from "node:assert/strict";
import { Cl } from "@stacks/transactions";

describe("test counter", () => {
  let session;
  before(async () => {
    session = await main();
    await session.initSession(process.cwd(), "./Clarinet.toml");
  });

  it("gets counter value", () => {
    let count = session.callReadOnlyFn({
      sender: "ST1SJ3DTE5DN7X54YDH5D64R3BCB6A2AG2ZQ8YPD5",
      contract: "counter",
      method: "get-counter",
      args: [],
    });

    assert.deepEqual(count.result, Cl.int(1))
  });
});

Currently the clarinet-sdk lib is kind of raw but the plan is to improve it to make it simpler and more user friendly (and opinionated).

When do you get back from vacation?

[friedger]

Here are some unit tests in clarity that should be supported as well Currently, using a clarinet extension:

Extension by @MarvinJanssen : https://github.com/Trust-Machines/stacks-sbtc/blob/cc4c5e9faf0233ed33839b0d7bb74c0b4bdf628c/sbtc-mini/ext/generate-tests.ts

Instructions how to use it: https://github.com/Trust-Machines/stacks-sbtc/tree/cc4c5e9faf0233ed33839b0d7bb74c0b4bdf628c/sbtc-mini#unit-testing

[moodmosaic]

In the context of the new clarinet-sdk, a possible integration with fast-check could look like this:

prop("ensures that <add> adds up the right amout", (n: UIntCV) => {
  const { result } = simnet.callPublicFn("counter", "add", [n], address1);
  expect(result).toBeOk(Cl.bool(true));

  const counter = simnet.getDataVar("counter", "counter");
  expect(counter).toBe(n);
}).check();

Note that n is generated (and shrinked) by fast-check. Then, the test would run by default 100 times without printing into the console unless there is a failure, in which case the shrinked counterexample is printed.

---

In addition to that, we can also customize/override the built-in arbitrary, for example:

prop("ensures that <add> adds up the right amount", (n: UIntCV) => {
  const { result } = simnet.callPublicFn("counter", "add", [n], address1);
  expect(result).toBeOk(Cl.bool(true));

  const counter = simnet.getDataVar("counter", "counter");
  expect(counter).toBe(n);
}).check({
  runs: 1000,
  logs: true,
  data: {
    n: { min: 123 },
  }
});

---

Or, explore a possible integration with @fast-check/vitest and provide a set of built-in arbitraries (where arbitrary is a pair of a generator and a shrinker):

prop([tx.UIntCV({ min: 0 })])(
  "ensures that <add> adds up the right amount",
  (n: UIntCV) => {
    const { result } = simnet.callPublicFn("counter", "add", [n], address1);
    expect(result).toBeOk(Cl.bool(true));

    const counter = simnet.getDataVar("counter", "counter");
    expect(counter).toBe(n);
  },
);

Of course, users are free to write and use their own arbitraries. This can cover pretty much all the scenarios for stateless property-based tests.

[hugocaillard]

@moodmosaic
Do you think this issue is still relevant?
Is it good enough to work with the sdk / vitest / fast-check?
Could we just provide some documentation or is there work to do on the clarinet side?

[moodmosaic]

@hugocaillard, the easiest step forward is to explore a possible integration with @fast-check/vitest and decide whether it makes sense to build a set of custom fast-check arbitraries for Clarity types:

prop([arb.UIntCV({ min: 0 })])(
  "ensures that <add> adds up the right amount",
  (n: UIntCV) => {
    const { result } = simnet.callPublicFn("counter", "add", [n], address1);
    expect(result).toBeOk(Cl.bool(true));

    const counter = simnet.getDataVar("counter", "counter");
    expect(counter).toBe(n);
  },
);

In this example, the (hypothetical) custom fast-check arbitrary arb.UIntCV({ min: 0 }) could have come from stacks.js, clarinet-sdk, or defined locally by the user. — This is good enough as a first step.

---

The approach I've taken with deno, before the creation of clarinet-sdk, was by just passing parameters to the tests and not have the user think about fast-check and arbitraries. — Here's how this could translate into clarinet-sdk:

prop("ensures that <add> adds up the right amount", (n: UIntCV) => {
  const { result } = simnet.callPublicFn("counter", "add", [n], address1);
  expect(result).toBeOk(Cl.bool(true));

  const counter = simnet.getDataVar("counter", "counter");
  expect(counter).toBe(n);
}).check({
  runs: 1000,
  logs: true,
  data: {
    n: { min: 123 },
  }
});

If any of the above matches with something we'd want to provide to the users, issue can remain open otherwise can be closed.