The fuzzer is not working

[Medowhill]

After the update, the fuzzer shows the following:

Traceback (most recent call last):
  File "tests/fuzz.py", line 252, in <module>
    fuzz(tests_dir, fuzz_arg, args.num)
  File "tests/fuzz.py", line 223, in fuzz
    raise Exception("Test `{}` failed with exit code {}.".format(" ".join(args), proc.returncode))
Exception: Test `cargo run --release --bin fuzz -- -p /Users/medowhill/workspace/rust/kecc-public/tests/test_polished.c` failed with exit code 101.

When I run cargo run --release --bin fuzz -- -p /Users/medowhill/workspace/rust/kecc-public/tests/test_polished.c, I can see the parser has panicked because of the unsupported features of KECC.

I tried again after revising fuzz.py as below, but string literals in generated code are problematic. I tried removing string literals, but it makes parsing fails.

-    "__restrict": "",
+    "restrict": "",
+    "_Nullable": "",
+    "inline ": "",
+    r"0x[0-9a-fA-F]*\.[0-9a-fA-F]*p\-?[0-9]*f?": "1.0f",
+    r"0x[0-9a-fA-F]+p\-?[0-9]*f?": "1.0f",
+    r"__asm\(.*\)": "",
     "long __undefined;": "",
     "return 0;": "return crc32_context % 128;",
-    r"__attribute__ \(\(.*\)\)": "",
+    r"__attribute__ ?\(\(.*\)\)": "",

As I am using macOS, I have not tried in Linux. Can it be a macOS-specific problem of csmith?

[ahcheongL]

The polished c code test_polished.c contains enum type which is not supported by KECC...
These are some code fragments that are in test_polished.c
line 1615-1621

enum __codecvt_result
{
  __codecvt_ok,
  __codecvt_partial,
  __codecvt_error,
  __codecvt_noconv
};

line 1258-1266

# 347 "/usr/include/math.h" 3 4
typedef enum
{
  _IEEE_ = -1,
  _SVID_,
  _XOPEN_,
  _POSIX_,
  _ISOC_
} _LIB_VERSION_TYPE;

I'm not sure what polished code is...
I'm using my own ubuntu server

[Medowhill]

Sorry, I was a bit wrong. String literals were not the source of problem. After I fixed my write_c.rs, with the revised fuzz.py as below, I could run the fuzzer.

     "volatile ": "",
     "static ": "",
     "extern ": "",
-    "__restrict": "",
+    "restrict": "",
+    "_Nullable": "",
+    "inline ": "",
+    r"0x[0-9a-fA-F]*\.[0-9a-fA-F]*p\-?[0-9]*f?": "1.0f",
+    r"0x[0-9a-fA-F]+p\-?[0-9]*f?": "1.0f",
+    r"__asm\(.*\)": "",
+    r"\".*\",": "",
     "long __undefined;": "",
     "return 0;": "return crc32_context % 128;",
-    r"__attribute__ \(\(.*\)\)": "",
+    r"__attribute__ ?\(\(.*\)\)": "",
     "_Float128": "long double",
     "union": "struct",
     r"enum\s*\{[^\}]*\};": "",

If it is not a macOS-specific problem, I believe that fuzz.py should be properly fixed.

[jeehoonkang]

Thank you for pointing out that. We're actively working on fixing the fuzzer. I promise we will upload a new vesion by the end of this week. Sorry again for the inconvenience.

[jeehoonkang]

It's now fixed in a recent commit: kaist-cp/kecc-public@9383908

[ahcheongL]

Still, I couldn't pass fuzz testing because the following code fragment with enum type was left in test_polished.c

# 347 "/usr/include/math.h" 3 4
typedef enum
{
  _IEEE_ = -1,
  _SVID_,
  _XOPEN_,
  _POSIX_,
  _ISOC_
} _LIB_VERSION_TYPE;
_LIB_VERSION_TYPE _LIB_VERSION;

This type was never used in the code, so by just adding the following replace string, I could pass the fuzz testing:
"typedef enum[\w\s]*\{[^\}]*\}[^;]*;[\s\w]*;": "",

Please consider adding this case.
Thanks.

[hyunsukimsokcho]

@LockOne Could you state in which environment (including kernel version) you are trying to run the fuzzer? We do some testing on ourselves (in provided server), however we couldn't encounter the problem. Indeed, it seems hard to reproduce the same error.

[ahcheongL]

Ubuntu 16.04.6 with gcc v5.4.0
It seems like an environment problem.

My write_c.rs passed gg.kaist.ac.kr's grading script, so it does not matter much,
I'm just worried about other students with different environments.
Thanks.

[hyunsukimsokcho]

@LockOne We appreciate not only reporting an issue but also suggesting a working patch. However, it seems removing the next line no matter what. (Although the sentence shall start with typedef enum ..) We are worried that it would remove meaningful line in the other environment. We try to be very careful with such ad-hoc solutions. I shall ping @jeehoonkang and ask if the patch is acceptable.

[j31d0]

As @Medowhill already pointed out in his suggestion, __asm should be removed in source code. But current latest commit does not handle this. When I face this issue, I also notice that in AssertSupported implementation for Declarator, they just call self.extension.is_empty() but it is just boolean value, so it does not raise panic even it has extensions. So, I suggest to patch AssertSupported for Declarator and ParameterDeclaration, and patch fuzzer to remove __asm extension.

diff --git a/src/c/parse.rs b/src/c/parse.rs
index be36ba6..19b6de3 100644
--- a/src/c/parse.rs
+++ b/src/c/parse.rs
@@ -217,7 +217,7 @@ impl AssertSupported for Declarator {
     fn assert_supported(&self) {
         self.kind.assert_supported();
         self.derived.assert_supported();
-        self.extensions.is_empty();
+        assert_eq!(true, self.extensions.is_empty());
     }
 }

@@ -282,7 +282,7 @@ impl AssertSupported for ParameterDeclaration {
     fn assert_supported(&self) {
         self.specifiers.assert_supported();
         self.declarator.assert_supported();
-        self.extensions.is_empty();
+        assert_eq!(true, self.extensions.is_empty());
     }
 }

diff --git a/tests/fuzz.py b/tests/fuzz.py
index f7ebcb2..1db1de3 100644
--- a/tests/fuzz.py
+++ b/tests/fuzz.py
@@ -36,6 +36,8 @@ REPLACE_DICT = {
     "\"g_\w*\", ": "",              # transparent_crc에서 프린트 목적으로 받은 StringLiteral 삭제
     "char\* vname, ": "",           # transparent_crc에서 사용하지 않는 파라미터 삭제
     r"[^\n]*_IO_2_1_[^;]*;": "",    # extern을 지우면서 생긴 size를 알 수 없는 struct 삭제
+    r"__asm\s*\([^\)]*\)": "",      # asm extension in mac
+    r"__asm__\s*\([^\)]*\)": "",    # asm extension in linux
 }
 CSMITH_DIR = "csmith-2.3.0"