urlscan Issues

[ssnkhan]

Hi there,

Seeing some unusual behaviour where all queries using urlscan refuse to work (when called via cron), generating the following error:

*A* `Mihari::Error` *occured in background*: Please configure Urlscan API settings properly
```/var/lib/gems/2.7.0/gems/mihari-1.5.1/lib/mihari/analyzers/base.rb:71:in `rescue in set_unique_artifacts'
/var/lib/gems/2.7.0/gems/mihari-1.5.1/lib/mihari/analyzers/base.rb:67:in `set_unique_artifacts'
/var/lib/gems/2.7.0/gems/mihari-1.5.1/lib/mihari/analyzers/base.rb:36:in `run'
/var/lib/gems/2.7.0/gems/mihari-1.5.1/lib/mihari/cli.rb:336:in `run_analyzer'
/var/lib/gems/2.7.0/gems/mihari-1.5.1/lib/mihari/cli.rb:55:in `block in urlscan'
/var/lib/gems/2.7.0/gems/mihari-1.5.1/lib/mihari/cli.rb:304:in `with_error_handling'
/var/lib/gems/2.7.0/gems/mihari-1.5.1/lib/mihari/cli.rb:54:in `urlscan'
/var/lib/gems/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
/var/lib/gems/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
/var/lib/gems/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'```

What I think is happening, is that urlscan the binary at usr/lib/urlscan is being executed, rather than the the mihari module itself. I have tried this on two separate machines and am getting the same error. The API key has been set correctly, and all other queries work without issue.

Oddly, running the query directly returns results as expected. I have tried a few things and struggling -- any ideas?

[ninoseki]

I just checked mihari urlscan command and it worked without the error.

As the error says, you have to set URLSCAN_API_KEY to use the command.

[ninoseki]

Sorry I overlooked this line.

The API key has been set correctly,

A possible cause is a typo.
Please make sure to set your urlscan.io API key as URLSCAN_API_KEY.
(ref. https://github.com/ninoseki/mihari/blob/master/lib/mihari/config.rb#L33)

[ssnkhan]

The key has been set correctly, as it will return results if typed directly in bash. But always fails when invoked with a cronjob (though all other queries work without issue).

[ssnkhan]

This must be a environmental issue - the cronjob works locally, but not in production.

[Canon5616]

This must be a environmental issue - the cronjob works locally, but not in production.

did you ever resolve this issue, i am having the same results. Runs with no issues when run manually but i get the following error when run via cron job:

023-06-16 00:10:06.073119 �[31mE�[0m [1725259:7440 error_notification.rb:12] �[31mMihari�[0m -- Exception: �[31mMihari::ConfigurationError: Urlscan is not configured correctly�[0m
/var/lib/gems/2.7.0/gems/mihari-5.2.3/lib/mihari/analyzers/rule.rb:197:in block in analyzers' /var/lib/gems/2.7.0/gems/mihari-5.2.3/lib/mihari/analyzers/rule.rb:183:in map'
/var/lib/gems/2.7.0/gems/mihari-5.2.3/lib/mihari/analyzers/rule.rb:183:in analyzers' /var/lib/gems/2.7.0/gems/mihari-5.2.3/lib/mihari/analyzers/rule.rb:244:in validate_analyzer_configurations'
/var/lib/gems/2.7.0/gems/mihari-5.2.3/lib/mihari/analyzers/rule.rb:53:in initialize' /var/lib/gems/2.7.0/gems/mihari-5.2.3/lib/mihari/structs/rule.rb:165:in new'
/var/lib/gems/2.7.0/gems/mihari-5.2.3/lib/mihari/structs/rule.rb:165:in analyzer' /var/lib/gems/2.7.0/gems/mihari-5.2.3/lib/mihari/commands/search.rb:41:in block in run'
/var/lib/gems/2.7.0/gems/mihari-5.2.3/lib/mihari/mixins/error_notification.rb:10:in with_error_notification' /var/lib/gems/2.7.0/gems/mihari-5.2.3/lib/mihari/commands/search.rb:40:in run'
/var/lib/gems/2.7.0/gems/mihari-5.2.3/lib/mihari/commands/search.rb:80:in block in search' /var/lib/gems/2.7.0/gems/mihari-5.2.3/lib/mihari/database.rb:170:in with_db_connection'
/var/lib/gems/2.7.0/gems/mihari-5.2.3/lib/mihari/commands/search.rb:62:in search' /var/lib/gems/2.7.0/gems/thor-1.2.2/lib/thor/command.rb:27:in run'
/var/lib/gems/2.7.0/gems/thor-1.2.2/lib/thor/invocation.rb:127:in invoke_command' /var/lib/gems/2.7.0/gems/thor-1.2.2/lib/thor.rb:392:in dispatch'
/var/lib/gems/2.7.0/gems/thor-1.2.2/lib/thor/base.rb:485:in start' /var/lib/gems/2.7.0/gems/mihari-5.2.3/exe/mihari:8:in <top (required)>'
/usr/local/bin/mihari:23:in load' /usr/local/bin/mihari:23:in

'

[ssnkhan]

@Canon5616 Sorry, I never got it to work.

[ninoseki]

There are two ways to set urlscan.io API key.

The first one is set it via URLSCAN_API_KEY environment variable. (You may not set it properly in cron)
The second one set it via api_key in a rule.

analyzer: urlscan
query: ...
api_key: your_api_key_goes_here

[Canon5616]

There are two ways to set urlscan.io API key.

The first one is set it via URLSCAN_API_KEY environment variable. (You may not set it properly in cron) The second one set it via api_key in a rule.

analyzer: urlscan
query: ...
api_key: your_api_key_goes_here

Thanks Ninoseki , adding in the api key to the yml file worked. Not sure why my cron job doesnt like the URLscan environment variable but is fine with the rest.

Thanks again!