Correlations issue

[dreathe]

Please provide a description of the issue and any relevant error messages.
Can't use the correlations, i have run with and without root permission against multiple scans, spiderfoot works well by itself i can browse through the data collected, i just cannot use the correlations:

python3 sf.py -C E30444D1
2022-04-28 23:47:45,486 [INFO] sf : Running 37 correlation rules against scan, E30444D1
...
Nothing happens no results.

If you can provide an application stack trace that is even better.

And this is the result after running the scan at the end after all the data being collected:

2022-04-28 06:39:20,148 [INFO] sflib : Running 37 correlation rules.
Process Process-1:
Traceback (most recent call last):
File "/home/luis/spiderfoot/spiderfoot/db.py", line 937, in scanResultEvent
self.dbh.execute(qry, qvars)
sqlite3.OperationalError: unable to open database file

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/luis/spiderfoot/sfscan.py", line 30, in startSpiderFootScanner
return SpiderFootScanner(*args, **kwargs)
File "/home/luis/spiderfoot/sfscan.py", line 221, in init
self.__startScan()
File "/home/luis/spiderfoot/sfscan.py", line 427, in __startScan
self.runCorrelations()
File "/home/luis/spiderfoot/sfscan.py", line 438, in runCorrelations
corr.run_correlations()
File "/home/luis/spiderfoot/spiderfoot/correlation.py", line 88, in run_correlations
results = self.process_rule(rule)
File "/home/luis/spiderfoot/spiderfoot/correlation.py", line 665, in process_rule
events.extend(self.collect_events(c['collect'],
File "/home/luis/spiderfoot/spiderfoot/correlation.py", line 388, in collect_events
events = self.collect_from_db(matchrule,
File "/home/luis/spiderfoot/spiderfoot/correlation.py", line 280, in collect_from_db
for row in self.dbh.scanResultEvent(**query_args):
File "/home/luis/spiderfoot/spiderfoot/db.py", line 940, in scanResultEvent
raise IOError(f"SQL error encountered when fetching result events: {e.args[0]}")
OSError: SQL error encountered when fetching result events: unable to open database file

The debug log shows, there are no correlations must be from the fact they are not being written in the first place.

What version of Python are you using?
Python 3.9.12

What version of SpiderFoot are you using (stable release or Git master branch)?
v4.0.0
You may also wish to check if your issue has been posted previously:

• https://github.com/smicallef/spiderfoot/issues
  spiderfoot.debug.log

[smicallef]

Could you check that SpiderFoot is not running already? The database might be locked by that process.

[dreathe]

when i launch spiderfoot it does indeed create 3 process running at the same time, tried to kill each one individual but it ends up exiting spiderfoot. Graphical interface does say "ABORT-REQUESTED" never switch to "Aborted" but it was been like that since release of v4.0.0 everytime i try to abort a scan.

$ ps aux | grep -e "sf"
root 194 0.0 0.0 0 0 ? I< 19:00 0:00 [ata_sff]
root 1395170 0.0 0.0 12504 5712 pts/0 S+ 23:35 0:00 sudo ./sf.py -l 127.0.0.1:9999
root 1395171 0.0 0.0 12504 2156 pts/1 Ss 23:35 0:00 sudo ./sf.py -l 127.0.0.1:9999
root 1395172 62.3 2.2 1164776 137440 pts/1 Sl+ 23:35 0:01 python3 ./sf.py -l 127.0.0.1:9999

[bcoles]

It looks like you have run SpiderFoot as root with sudo.

Have you treid running the correlations with sudo as well?

sudo python3 sf.py -C E30444D1

Graphical interface does say "ABORT-REQUESTED" never switch to "Aborted" but it was been like that since release of v4.0.0 everytime i try to abort a scan.

I'm not entirely sure what you mean here. SpiderFoot correlations work on aborted scans in ABORT-REQUESTED state. A scan should never enter ABORT-REQUESTED state unless the scan was manually aborted via the web interface.

[dreathe]

@bcoles Thanks It works if i run as sudo individually what i meant was the state never changes, it used to change after it actually aborted to Abort. Now in v4.0.0 it always maintain the ABORT-REQUESTED state in the web display even after it stopped.

I have deleted my old saves that i had aborted and run it against the scan it marked as completed. Not sure going to try more, but atleast now i can use the correlation, so im happy already :D

[bcoles]

Can't use the correlations [...] i can browse through the data collected, i just cannot use the correlations:

Presumably this is resolved.

And this is the result after running the scan at the end after all the data being collected:
sqlite3.OperationalError: unable to open database file

Presumably this is resolved.

it used to change after it actually aborted to Abort. Now in v4.0.0. Now in v4.0.0 it always maintain the ABORT-REQUESTED state in the web display even after it stopped.

You are saying that every aborted scan enters ABORT-REQUESTED state and never enters ABORTED state? How were you aborting the scans? If you force kill spiderfoot (or hit ctrl+c) it won't get a chance to update the state.

[bcoles]

I wasn't able to reproduce this issue. Starting a scan then manually stopping the scan before the scan completed resulted in the scan state changing to ABORT-REQUESTED. Once all running threads finished the scan state changed to ABORTED. Note that the scan state won't change in the web UI while you're viewing a scan. You will need to browse to the Summary tab of the scan or browse back to the scan list.

[dreathe]

You are saying that every aborted scan enters ABORT-REQUESTED state and never enters ABORTED state? How were you aborting the scans? If you force kill spiderfoot (or hit ctrl+c) it won't get a chance to update the state.

Yes i just tried this again, aborted using the web interface, scan then stopped, nothing happens still says "ABORT-REQUESTED", started another scan. Still says the same "ABORT-REQUESTED" after the new one started.

Going to try again to run a scan, and see if it runs the correlations in the end without the need to do it manually.

But i did try to run the correlations against a aborted scan and didnt work, but again i was using it the wrong way like u said. So i dont know until i try again, im going to let this one finish and see if it does the correlations.

Thank for all the help u guys are great. Cya tomorrow or in 2 days when the scan ends 👍

[bcoles]

im going to let this one finish and see if it does the correlations.
Thank for all the help u guys are great. Cya tomorrow or in 2 days when the scan ends

That seems like a painful way to test. Why not run a short scan instead?

[Asiacalbear2020]

I've unsubscribed twice in the past and still getting these emails. PLEASE remove me from this thread. Thank you.

[bcoles]

I've unsubscribed twice in the past and still getting these emails. PLEASE remove me from this thread. Thank you.

There's nothing anyone can do to unsubscribe you from GitHub emails. You will need to modify your GitHub settings. Or, if you wish to stop receiving emails about the SpiderFoot project you can adjust the "watch" settings for this repository:

[dreathe]

Sorry for the late response, everything seems to be working as expected.
I would like to provide you guys with a better explanation, but everything is working now, i haven't done anything different but the correlations are working after the scan has stopped. Individual scans are working as well, and the scans that previously showed as ABORT-REQUESTED are now showing Aborted like expected.

So everything is working as far as this issue was initially open. So to me is closed 👍

I have other "stupid" questions about errors during the scan execution do u guys have a slack channel or anything like that ?

Thanks again for all the good support with the issues encountered on my side.

[bcoles]

I have other "stupid" questions about errors during the scan execution do u guys have a slack channel or anything like that ?

There's a discord. https://discord.gg/vyvztrG