Add fuzz testing local step and CI integration #1768

jeffreylovitz · 2021-06-11T18:24:16Z

Run fuzz test for 30 seconds in CI and adds make fuzz rule.

The test fails if the server crashes during its execution.

swilly22

This is great, next step would be running against a populated graph and increasing the chances of generating a query which will pass query validations.

swilly22 · 2021-06-13T11:12:23Z

.circleci/config.yml

+      - run:
+          name: Run 30-second fuzz test
+          command: |
+              make fuzz # Run the fuzz tester


looking at CI it seems like make fuzz build RedisGraph, please try to run the fuzzer on the artifact of the build step, consult with either @chayim or @rafie if you need help here.

chayim · 2021-06-14T08:37:21Z

tests/requirements.txt

@@ -3,3 +3,4 @@ redisgraph>=1.5
 redisgraph-bulk-loader
 click>=6.7
 behave>=1.2
+git+https://github.com/jeffreylovitz/grammarinator.git@master


Rather than rely on master pin to the hash.

chayim · 2021-06-14T08:40:20Z

tests/fuzz/process.py

+    start = time()
+    while time() - start < timeout:
+        # Capture generated queries
+        query = subprocess.check_output(cmd, env=env, shell=True, encoding="utf8")


Catch and handle the CalledProcessError if check_output fails?

chayim · 2021-06-14T08:41:49Z

tests/fuzz/process.py

+
+
+def issue_queries(graph, timeout):
+    working_dir = os.path.dirname(os.path.abspath(__file__))


Style: wrap abspath around dirname instead of dirname around abspath since dirname just removes the last segment of the path. It's more stable on symlinked directories.

chayim

Great stuff, small comments only!

swilly22 · 2021-06-15T05:34:11Z

@jeffreylovitz please look into CI build failure

codecov · 2021-06-18T19:56:24Z

Codecov Report

❗ No coverage uploaded for pull request base (master@6551611). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 78b7f37 differs from pull request most recent head 19bcd0e. Consider uploading reports for the commit 19bcd0e to get more accurate results

@@            Coverage Diff            @@
##             master    #1768   +/-   ##
=========================================
  Coverage          ?   85.86%           
=========================================
  Files             ?      230           
  Lines             ?    21011           
  Branches          ?        0           
=========================================
  Hits              ?    18042           
  Misses            ?     2969           
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6551611...19bcd0e. Read the comment docs.

* Add fuzz testing local step and CI integration * Address PR comments * Use upstream repo of Grammarinator Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> (cherry picked from commit ec7fdb6)

* update RE QA versions (#1750) * Support updating properties with map referenced by parameter or variable (#1749) * Support updating properties with map referenced by parameter or variable * Unify logic for updating property maps * refactor, simplified update construction logic * reply with error on cleanup * tidy up Co-authored-by: swilly22 <roi@redislabs.com> * circleci context (#1753) access to the variables in the organization context * map should return volatile data (#1754) * Reduce edge count optimization 3 (#1756) * Use the statistics to determine edge count. * amend to prev. * Added edge_count optimization benchmark * benchmark both node and edge count Co-authored-by: swilly22 <roi@redislabs.com> * Fix edge_count.yml benchmark name and init_commands (#1765) * updating to redis 6.2.4 (#1762) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * [fix] Added missing common context to nightly perf automation (#1770) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * rce package (#1775) * Updated readies and CircleCI config (#1774) * Updated readies * fixes 1 * fixes 2 * fixes 3 * fixes 4 * Coverage analysis (#1777) * [WIP] Coverage analysis * fixes 1 * fixes 2 * fixes 3 * fixes 4 Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * fixed bug in load time config in which config subscribers missed them… (#1764) * fixed bug in load time config in which config subscribers missed them and also enabled run time configuration of multiple fields at once. * refactor * amend to prev. * amend to prev * amend to prev * refactor configuration dryrun * validate configuration wasn't changed * amend to prev - extended the test * init thread pools Co-authored-by: swilly22 <roi@redislabs.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * array refactor - use do while(0) instead of compound expression. (#1776) * array refactor - use do while(0) instead of compound expression. * array_append - refactor, change to do while(0) * indentation Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: swilly22 <roi@redislabs.com> * matrix resize doesn't set both rows and columns atomicly (#1773) * Add fuzz testing local step and CI integration (#1768) * Add fuzz testing local step and CI integration * Address PR comments * Use upstream repo of Grammarinator Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * multi property index support. (#1782) * multi property index support. * amend to prev * amend to prev. Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * use individual edge deletion when number of edges to delete is small (#1783) * use individual edge deletion when number of edges to delete is small * Add edge deletion benchmark Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> * On CondVarLenTraverseOptimizedConsume init allNeighborsCtx instead of… (#1784) * On CondVarLenTraverseOptimizedConsume init allNeighborsCtx instead of realloc on each iteration. * reset assume context already created Co-authored-by: swilly22 <roi@redislabs.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Add documentation for startNode() and endNode() (#1792) * Fix memory leak on AllNeighborsCtx_Reset (#1791) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Add 10-minute nightly fuzz test (#1790) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * CentOS8/RHEL8 builds (#1797) * CentOS8/RHEL8 builds * fixes 1 * fixes 2 * fixes 3 * fixes 4 * fixes 5 * Don't use bulk routine for implicit edge deletion <= 4 (#1793) * Don't use bulk routine for implicit edge deletion <= 4 * avoid tuple iterator, switch to matrix export * Minimize number of Apply calls in deletion * Explicitly label node in Graph_CreateNode, fix memory leaks Co-authored-by: swilly22 <roi@redislabs.com> * unzipping just the !rce now that there are dupes (#1803) * removing docker curl and executing, relying on the contents of github directly (#1802) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * setting matrix sparse on new relationship type. (#1799) * setting matrix sparse on new relationship type. * amend to prev * amend to prev Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * perf clone fix (#1807) * AllNeighborsCtx should hold array of GxB_MatrixTupleIter and not arra… (#1794) * AllNeighborsCtx should hold array of GxB_MatrixTupleIter and not array of GxB_MatrixTupleIter* * amend to prev * amend to prev. * Update GxB_Matrix_tuple_iter.c Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Tuple iter race (#1788) * Resolve race condition in GxB_MatrixTupleIter * Temporarily modify GraphBLAS README * Always set sparsity on RG_Matrix objects * set sparsity control on creation * tuple iterator set sparsity control * Disable GraphBLAS build parallelism Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: swilly22 <roi@redislabs.com> * Add codecov badge (#1789) Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * adding more indexes to test_persistency (#1760) * adding more indexes to test_persistency * amend to prev * refactor test Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> Co-authored-by: swilly22 <roi@redislabs.com> * Build: allow specification of parallel jobs (#1809) By default, do not exceed 8 jobs, so we don't OOM in CI. Override (e.g. make JOBS=32) is possible. Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * fixed aggregate ambigous grouping. (#1787) * fixed aggregate ambigous grouping. * add group_hash * fixed a bug in prev commit. * Update op_aggregate.c Co-authored-by: swilly22 <roi@redislabs.com> Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * fixing peg to install man pages into local tree (#1815) Given that we don't use the man pages, but the make install target requires it, this ensures it won't break, and doesn't pollute the filesystem * Arm platform build (#1810) * Arm platform build * fixes 1 * fixes 2 * fixes 3 * fixes 4 Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Release drafter (#1800) * Create release-drafter-config.yml * Create release-drafter.yml * Create codeql-analysis.yml (#1795) * Create codeql-analysis.yml * Update codeql-analysis.yml * Update codeql-analysis.yml * Update codeql-analysis.yml Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com> * Fix build issue on macos 11 and treat misleading indentation as warning (#1820) * fix build * fix build Co-authored-by: Jeffrey Lovitz <jeffrey.lovitz@gmail.com> Co-authored-by: Chayim <chayim@users.noreply.github.com> Co-authored-by: OfirMos <82366525+OfirMos@users.noreply.github.com> Co-authored-by: filipe oliveira <filipecosta.90@gmail.com> Co-authored-by: Rafi Einstein <raffapen@outlook.com> Co-authored-by: Guy Korland <gkorland@gmail.com> Co-authored-by: Avi Avni <avi.avni@gmail.com>

* Add fuzz testing local step and CI integration * Address PR comments * Use upstream repo of Grammarinator Co-authored-by: Roi Lipman <swilly22@users.noreply.github.com>

jeffreylovitz added the enhancement label Jun 11, 2021

jeffreylovitz requested a review from swilly22 June 11, 2021 18:24

swilly22 previously approved these changes Jun 13, 2021

View reviewed changes

chayim reviewed Jun 14, 2021

View reviewed changes

chayim suggested changes Jun 14, 2021

View reviewed changes

jeffreylovitz dismissed swilly22’s stale review via a1b1621 June 14, 2021 14:44

jeffreylovitz force-pushed the fuzz-testing branch 3 times, most recently from b0c4c41 to f903f9d Compare June 14, 2021 16:32

jeffreylovitz added 3 commits June 18, 2021 15:21

Add fuzz testing local step and CI integration

ed3bd4d

Address PR comments

478503d

Use upstream repo of Grammarinator

78b7f37

jeffreylovitz force-pushed the fuzz-testing branch from bcfc6e6 to 78b7f37 Compare June 18, 2021 19:23

Merge branch 'master' into fuzz-testing

19bcd0e

swilly22 approved these changes Jun 19, 2021

View reviewed changes

swilly22 merged commit ec7fdb6 into master Jun 19, 2021

swilly22 deleted the fuzz-testing branch June 19, 2021 06:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add fuzz testing local step and CI integration #1768

Add fuzz testing local step and CI integration #1768

jeffreylovitz commented Jun 11, 2021 •

edited

Loading

swilly22 left a comment

swilly22 Jun 13, 2021

chayim Jun 14, 2021

chayim Jun 14, 2021

chayim Jun 14, 2021

chayim left a comment

swilly22 commented Jun 15, 2021

codecov bot commented Jun 18, 2021 •

edited

Loading



		def issue_queries(graph, timeout):
		working_dir = os.path.dirname(os.path.abspath(__file__))

Add fuzz testing local step and CI integration #1768

Add fuzz testing local step and CI integration #1768

Conversation

jeffreylovitz commented Jun 11, 2021 • edited Loading

swilly22 left a comment

Choose a reason for hiding this comment

swilly22 Jun 13, 2021

Choose a reason for hiding this comment

chayim Jun 14, 2021

Choose a reason for hiding this comment

chayim Jun 14, 2021

Choose a reason for hiding this comment

chayim Jun 14, 2021

Choose a reason for hiding this comment

chayim left a comment

Choose a reason for hiding this comment

swilly22 commented Jun 15, 2021

codecov bot commented Jun 18, 2021 • edited Loading

Codecov Report

jeffreylovitz commented Jun 11, 2021 •

edited

Loading

codecov bot commented Jun 18, 2021 •

edited

Loading