Provide a way to use scanpipe with an in-process database

[cco3]

Depending on the environment, using Postgres can be a little unwieldy (requires a daemon, root access, etc.).

Is the DB used for much more than just storing pipeline statuses and results? Is there any reason a solution like SQLite couldn't work here?

[tdruez]

No particular reasons, SQLite should work mostly fine for a single user usage, although it hasn't been extensively tested.
See https://docs.djangoproject.com/en/3.1/ref/databases/#sqlite-notes for the SQLite limitations.

I will provide a simple way to setup SQLite over PostgreSQL with documentation.
In the mean time you can already try a SQLite setup it with the following steps:

• After the make envfile do not run the make cleandb, instead:
• Edit the scancodeio/settings/base.py and locate the DATABASES setting
• Replace "ENGINE": "django.db.backends.postgresql" with "ENGINE": "django.db.backends.sqlite3"
• Replace "NAME": os.getenv("SCANCODEIO_DB_NAME", "scancodeio") with "NAME": "sqlite3.db"
• Run make migrate

In summary:

git clone git@github.com:nexB/scancode.io.git && cd scancode.io
make dev
make envfile
[Edit the settings]
make migrate

We'll add a new make sqlite command that will take care of the settings and the migration.

[tdruez]

@cco3 PR at #57
Let me know it that approach works for you.

[pombredanne]

@tdruez excellent!
@cco3 there is no specific dependency on PostgreSQL at this stage since the latest versions of Django to support JSON fields on all supported DB backends including SQLlite.
Things are not tested yet on SQLlite but are likely to work OK
But you need to be aware that there may be issues with access contention whenever you run more than one pipeline at a time or have more than one project and also when we will start to parallelize specific pipeline steps for performance (soon enough). This will likely break SQLite because of locking so you need to keep this in mind.

[pombredanne]

@tdruez re: #56 (comment)

Edit the scancodeio/settings/base.py and locate the DATABASES setting

Would it be possible to have a local setting file that's not the main version-controlled one?
I created #58

[tdruez]

@pombredanne this edit in base.py in not needed anymore, this was just a temporary workaround.
PR #57 allows to override those settings directly in the .env file.