approach/setting to control search space of a specific attribute

[jakc4103]

Is there any approach/setting to control the search space for value of some specific OP attributes?
For example, any approach to control kernel_size of Conv2d to be in range [1, 15]?

I tried to generate a model with conv2d and relu only, with max_nodes set to 100.
If the timeout_ms was set to 10000 (relativly small number) ms, the solver seems to find nothing, then model_gen failed with following errors.
If timeout_ms set to 3000000 (a large number) ms, the entire model_gen process takes a very long time to generate a single model

  File "/venv/test/bin/nnsmith.model_gen", line 8, in <module>
    sys.exit(main())
  File "/venv/test/lib/python3.8/site-packages/hydra/main.py", line 90, in decorated_main
    _run_hydra(
  File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 222, in run_and_report
    raise ex
  File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 219, in run_and_report
    return func()
  File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/venv/test/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/venv/test/lib/python3.8/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/venv/test/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/venv/test/lib/python3.8/site-packages/nnsmith/cli/model_gen.py", line 57, in main
    f"{len(gen.last_solution)} symbols and {len(gen.solver.assertions())} constraints."
TypeError: object of type 'NoneType' has no len()

[lazycal]

For example, any approach to control kernel_size of Conv2d to be in range [1, 15]?

One ad-hoc way would be to add something like cons.append(nnsmith_le(kernel_h_size, 15)) around here

nnsmith/nnsmith/abstract/op.py

Line 1424 in 7b793ff

cons.append(nnsmith_ge(mimic_kh, 1))

[ganler]

Recall that conv2d only accepts 4-dimension inputs. Because the graph generation is randomized in a way to start with a placeholder that has a random rank, as a result, it must be lucky enough to start with 4 to be able to generate conv2d.

This should be fixed by adding an extra argument for configuring the rank (length of its shape dimensions) of the starter placeholder (WIP).

For adding extra constraints, because we don't want to bother users by editing the source code, there is a way to systematically patch constraints at user level with the patch_requires decorator. See the TensorRT example:

nnsmith/nnsmith/backends/tensorrt.py

Line 151 in bdc2747

@patch_requires(TRTFactory.system_name, "core.Pool2d")

Probably something like:

@patch_requires(${THE_FACTORY_TYPE}$.system_name, "core.Pool2d")
def RulePool2d(self: AbsOpBase, _: List[AbsTensor]) -> List[Union[z3.BoolRef, bool]]:
    return [self.kernel_h_size <= 15, self.kernel_w_size <= 15]

Meanwhile for some good randomness of shapes, try to use other method="symbolic-cinit" in model_gen:

gen = model_gen(
    ...
    method: str = "symbolic-cinit", # or "concolic"
    ...
):

My apologies for the incomplete documentation and will be improving the doc soon (prob. next mon). For the rank issue there will be a patch soon and I will offer a colab example later today.

Meanwhile, for now please try to use the latest version with:

pip install "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade

This will provide a lot more features and possibly a better experience (in debugging etc.). Thanks!

[ganler]

@jakc4103 You can try https://github.com/ise-uiuc/nnsmith/blob/main/doc/cli.md#add-extra-constraints with the latest nnsmith:

pip install "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade

[ganler]

Also, you might find the gist helpful: https://colab.research.google.com/drive/13LNQBvfpPFiaHWKnac6hxzvJY4LUAWWQ?usp=sharing

[jakc4103]

@jakc4103 You can try https://github.com/ise-uiuc/nnsmith/blob/main/doc/cli.md#add-extra-constraints with the latest nnsmith:

pip install "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade

Tks, this works.

[ganler]

Feel free to reopen if you still encounter any issues on this.