GQS has detected 36 bugs in Neo4j, Memgraph, FalkorDB, and Kuzu in total. 26 of them are logic bugs, and 10 of them are other bugs, like database crashes or unhandled exceptions. The bug list is as follows:
The test cases involve 32 functions, including:
{'min', 'collect', 'substring', 'CAST', 'sum', 'size', 'toInteger', 'replace', 'cos', 'tan', 'right', 'reverse', 'abs', 'max', 'acos', 'ltrim', 'sqrt', 'toBoolean', 'lTrim', 'e', 'upper', 'type', 'toLower', 'split', 'count', 'exp', 'endNode', 'suffix', 'floor', 'rTrim', 'toUpper', 'starts_with'}
To build GQS, please make sure that you have the following environment installed:
Maven 3.9.6
Java-JDK 21.0
Then, use the following command to compile the source code:
mvn install -DskipTests -T1C
You will get the executable JAR file at "targets/"
GQS is implemented based on Java.
To run the GQS, please make sure that you have correctly installed Java 21.0.
The GQS requires a database mode parameter specifying the database you want to test. Currently, GQS supports the following database mode:
neo4j -> Neo4j
memgraph -> Memgraph
falkordb -> FalkorDB
kuzu -> Kùzu
Before running GQS, please correctly configure the config.txt, which specifies the commands to start, stop and reset the database system. GQS executes these commands every time a new test cycle starts.
In the following, we describe the configuration process of each database in detail.
- Download the Neo4j source code from the Github repository (https://github.com/neo4j/neo4j).
- Compile and unzip the compiled compression file to get the following folder structure. Please put the
change_port.shin our repository into the folder. This script will change the port configuration of Neo4j to make sure GQS can correctly connect to the database under test.
├── bin
├── certificates
├── change_port.sh // From our repository
├── conf // Configuration files to adjust
├── data
├── import
├── labs
├── lib
├── licenses
├── LICENSES.txt
├── LICENSE.txt
├── logs
├── NOTICE.txt
├── packaging_info
├── plugins
├── README.txt
├── run
└── UPGRADE.txt
- Edit the configuration
conf/neo4j.conf. Change the following item:
server.bolt.listen_address=127.0.0.1:___PORT___1
server.http.listen_address=127.0.0.1:___PORT___2
- Configure the
config.txtbased on the directory of your Neo4j database. The following commands give you an example when you install the Neo4j database inneo4jfolder at the current directory, and place testing temporary files in~/neo4j/. Please note thatTHREAD_FOLDERis a constant that will be substituted by GQS to put databases under different testing threads to distinct locations.THREAD_WEBandTHREAD_SERVERwill be substituted by GQS to specify the port.LOG_DIRECTORYwill be changed to the corresponding log folder.
startCommand=aa=$PWD; mkdir -p ~/neo4j/THREAD_FOLDER; cd ~/neo4j/THREAD_FOLDER; cp -r $aa/neo4j ~/neo4j/THREAD_FOLDER; mkdir -p ./logs/neo4j; cd neo4j; ./change_port.sh conf/neo4j.conf THREAD_WEB THREAD_SERVER; ./bin/neo4j-admin server console 2>&1 &
stopCommand=kill -9 `netstat -tulnp | grep :THREAD_WEB | awk '{print $7}' | cut -d'/' -f1`
resetCommand=rm -rf ~/neo4j/THREAD_FOLDER
Please make sure that the port 20000 is not occupied by any other processes. Use the following command to initiate the testing process:
java -jar GQS-1.0-SNAPSHOT.jar neo4j
You will find the testing log and results in logs folder.
- Download the Memgraph source code from the Github repository (https://github.com/memgraph/memgraph).
- Compile the source code and copy the Memgraph database binary into the current directory.
- Configure the
config.txtbased on your needs. The following commands give you an example of when you put the Memgraph binary at the current folder, and test temporary files in~/memgraph/. Please note thatTHREAD_FOLDERis a constant that will be substituted by GQS to put databases under different testing threads in distinct locations.THREAD_WEBwill be substituted by GQS to specify the port.
startCommand=aa=$PWD; mkdir -p ~/memgraph/THREAD_FOLDER; cd ~/memgraph/THREAD_FOLDER; mkdir -p ./logs/memgraph; cp $aa/memgraph ~/memgraph/THREAD_FOLDER; cp $aa/libmemgraph_module_support.so ~/memgraph/THREAD_FOLDER/; ./memgraph --bolt-address 127.0.0.1 --bolt-port THREAD_WEB --storage-properties-on-edges true --query-execution-timeout-sec=30 > LOG_DIRECTORY/THREAD_FOLDER.log 2>&1 &
stopCommand=kill -9 `netstat -tulnp | grep :THREAD_WEB | awk '{print $7}' | cut -d'/' -f1`;rm -rf ~/memgraph/THREAD_FOLDER
resetCommand=kill -9 `netstat -tulnp | grep :THREAD_WEB | awk '{print $7}' | cut -d'/' -f1`;rm -rf ~/memgraph/THREAD_FOLDER
Please make sure that the port 20000 is not occupied by any other processes. Use the following command to initiate the testing process:
java -jar GQS-1.0-SNAPSHOT.jar memgraph
You will find the testing log and results in logs folder.
- Download the FalkorDB source code from the Github repository (https://github.com/FalkorDB/FalkorDB).
- Compile the source code and copy the suitable
falkordb.solibrary into the directory where redis server is installed, configure the redis server to automatically load this module once started. - Configure the
config.txtbased on your needs. The following commands give you an example of when you put the redis server and the falkordb library into thefalkordbfolder in the current directory, and test temporary files in~/falkordb/. Make sure you also putchange_port.shinto the Redis server folder to enable GQS to automatically change the port redis server binds to. Please note thatTHREAD_FOLDERis a constant that will be substituted by GQS to put databases under different testing threads in distinct locations.THREAD_WEBwill be substituted by GQS to specify the port.
startCommand=aa=$PWD/falkordb/; mkdir -p ~/falkordb/THREAD_FOLDER; cd ~/falkordb/THREAD_FOLDER; mkdir -p ./logs/falkordb/; cp $aa/* ~/falkordb/THREAD_FOLDER; ./change_port.sh redis.conf THREAD_WEB THREAD_WEB; ./redis-server ./redis.conf > LOG_DIRECTORY/THREAD_FOLDER.log 2>&1 &
stopCommand=kill -9 `netstat -tulnp | grep :THREAD_WEB | awk '{print $7}' | cut -d'/' -f1`
resetCommand=rm -rf ~/falkordb/THREAD_FOLDER
- Revise the redis server configuration
redis.conf, specifically in the following item.
port ___PORT___1
Please make sure that the port 20000 is not occupied by any other processes. Use the following command to initiate the testing process:
java -jar GQS-1.0-SNAPSHOT.jar falkordb
You will find the testing log and results in logs folder.
The Kuzu database is already embedded in the GQS. You do not need to download or compile it, but just need to correctly configure the config.txt. By default, the temporary testing files will be put at ~/kuzu.
startCommand=random
stopCommand=random
resetCommand=rm -rf ~/kuzu/THREAD_FOLDER
Use the following command to initiate the testing process:
java -jar GQS-1.0-SNAPSHOT.jar kuzu
You will find the testing log and results in logs folder.