Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shortly run SQLancer on the individual DBMS as part of the CI #3

Closed
mrigger opened this issue May 26, 2020 · 2 comments
Closed

Shortly run SQLancer on the individual DBMS as part of the CI #3

mrigger opened this issue May 26, 2020 · 2 comments
Labels

Comments

@mrigger
Copy link
Contributor

mrigger commented May 26, 2020

It would be desirable to add smoke testing to the CI, to verify that the testing implementation for one of the DBMS is not obviously broken.

@mrigger mrigger added the ci label May 26, 2020
@mrigger
Copy link
Contributor Author

mrigger commented Jun 23, 2020

This seems to be more tricky than initially thought. For DuckDB and SQLite, running SQLancer might cause the JVM to crash when a memory error is triggered (see #1). Also for the other DBMS, it might be that bugs are still triggered on their latest version, either since a new bug was introduced or because an old one has not yet been fixed. An option would be to disable checking whether SQLancer finds a bug, but then we would also not detect any regression bugs in SQLancer.

I think the optimal setup would be if the DBMS under test would, for each commit to it, pull the latest (release) version of SQLancer and run it for a while (e.g., 10 minutes), to prevent that any new bugs are introduced to the DBMS. SQLancer could, in turn, test the latest (release) version of the DBMS to check that SQLancer is not obviously broken.

@mrigger
Copy link
Contributor Author

mrigger commented Jul 16, 2020

Closing this, since SQLancer now runs tests for the DBMS it supports.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant