Alert.pl issues on Ubuntu 16.04 - SCOT 3.5.2

[mwarren921]

Anyone have any idea how to fix this issue with Alert.pl? I installed SCOT 3.5.2 from the repository on Ubuntu 16.04. When Alert.pl is run from any user (e.g. SCOT); I get this error which is right after "sender is approved" on messages which should be parsed and ingested. We are installing this new SCOT with "local" authorization at this time.

Here is the error:
Can't locate object method "parser_dir" via package "Scot::Env" at /opt/scot/lib/Scot/App/Mail.pm

Anyone have any idea as to how to fix this issue? Let me know if any additional details are needed.

Thanks,

Matt

[toddbruner]

Matt:

Thanks for the report. Looks like I forgot to put the parser_dir attribute in the /opt/scot/etc/alert.cfg.pl sample file. Try inserting the following line at line 2.

parser_dir => '/opt/scot/lib/Scot/Parser',

I'll be adding it to the sample soon, but you don't have to wait on that to try it out. Save the config file with the new line and at the next run of /opt/scot/bin/alert.pl you will hopefully get past that error.

[mwarren921]

Thanks Todd. I will get that in there and let you know if that works. Matt From: Todd Bruner [mailto:notifications@github.com] Sent: Thursday, March 09, 2017 1:00 PM To: sandialabs/scot Cc: Warren, Matthew D; Author Subject: Re: [sandialabs/scot] Alert.pl issues on Ubuntu 16.04 - SCOT 3.5.2 (#38) Matt: Thanks for the report. Looks like I forgot to put the parser_dir attribute in the /opt/scot/etc/alert.cfg.pl sample file. Try adding the following line at line 2. parser_dir => '/opt/scot/lib/Scot/Parser', I'll be adding it to the sample soon, but you don't have to wait on that to try it out. Save the config file with the new line and at the next run of /opt/scot/bin/alert.pl you will hopefully get past that error. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#38 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ASrBJn22DjfuS6TXoA2c8UzTMcoeMaUsks5rkGhAgaJpZM4MW8G7>.

[mwarren921]

I added that line and am getting this error now:

Can't located object method "mq" via package "Scot::Env" at /opt/scot/lib/Scot/Collection/Alertgroup.pm line 37.

Then it looked like it hung up and dumped some issues related to IMAPClient.pm line 122. Looks like it just lost connection to the server.

Matt

[toddbruner]

Apologies, the config system was the last thing we updated and our sample configs appear to be somewhat incomplete. Quick fix is to go into /opt/scot/etc/scot.cfg.pl and in the "modules" array, copy the "mq" section to /opt/scot/etc/alert.cfg.pl. modules array.

In other words, put:

{
        attr    => 'mq',
        class   => 'Scot::Util::Messageq',
        config  => {
            destination => "scot",
            stomp_host  => "localhost",
            stomp_port  => 61613,
        },
    },

in alerts.cfg.pl right after the line

modules => [

Thanks for the update, I'll be fixing the configs soon.

[mwarren921]

Hi Todd,
There were a few more config hurdles which I resolved. It actually processed 4 alerts then died again with an error: Can't call method "from_addresses" on an undefined value at /opt/scot/lib/Scot/Util/Imap.pm

What can I do to get it past this? Thanks.

[toddbruner]

Matt:

Please include the line numbers if possible, that will help with debugging. (Also sanitized sections of the logs in /var/log/scot will help)

In this case though, the error is indicating that the imap server was not able to get the "envelope" of the email message. If you look through /var/log/scot/scot.mail.log you might see a line containing: "Error from IMAP: " I'd be interested in see what that says. Also you might change "log_level" in etc/alert.cfg.pl to "TRACE" to get a few more details in the log.

I'd also be interested if you mark message 4 as read, will alert.pl continue processing message 5 etc. That would let us know if there is something funky with that message.

[mwarren921]

Hi Todd,
Sorry that line was 497 in the Imap.pm. So the full error was:
Can't call method "from_addresses" on an undefined value at /opt/scot/lib/Scot/Util/Imap.pm line 497.

I turned on TRACE in the alert config to see if I can get more details. Looking at the scot.mail.log file; It looked like it was dying on a large HTML formatted Email coming from Splunk. I have a FAILED to process error which came from Mail.pm and it stuck the whole Email into a $VAR1. These Emails are still getting processed in the older version of SCOT.

Matt

[mwarren921]

Todd,
I cleared out all the Emails for the day where it was dying and it seems to have resolved that. I am on to the next issue which is the ElasticSearch cluster. While it looks to be running I am seeing "500" errors with Firebug when trying to do a search.

"NetworkError: 500 Internal Server Error - https://xxxxxx.xxx.xxx/scot/api/v2/search?qstring=xx 500 Internal Server Error.

I restarted the Elasticsearch service and that didn't resolve it. I will continue to hunt this one down.

Thanks,

Matt

[mwarren921]

Todd,
Concerning the elasticsearch issue if I copy out that URL and run it I see the mojolicious stack trace which is pointing to another issue with the "Scot::Env" Here is the error:

Can't locate object method "es" via package "Scot::Env" at /opt/scot/script/.../lib/Scot/Controller/Search.pm line 83.

Line 83 is: my $esua = $env->es;

Thanks,

Matt

[mwarren921]

Hi Todd,
I figured I needed to add a module for the Elasticsearch in the scot.cfg.pl. I pulled that from the stretch.cfg.pl and added it to scot.cfg.pl.

So I added this to the modules section (scot.cfg.pl.):

    **{
        attr    => 'es',
        class   => 'Scot::Util::ElasticSearch',
        config  => {
            nodes   => [ qw(localhost:9200) ],
        },
    },**

The environment error went away but it came back with a new error:

Can't locate object method "do_request_new" via package "Scot::Util::ElasticSearch" at /opt/scot/script/../lib/Scot/Controller/Search.pm line 94.

Please let me know if I was on the right track with what I did and if there is something I can do to fix this new error.

Thanks,

Matt

[toddbruner]

OK, first let me apologize profusely. All these problems stem from poorly tested sample configs. I appreciate your patience and perseverance. Replace the "es" section you copied with the following:

{
        attr    => 'es',
        class   => 'Scot::Util::ESProxy',
        config  => {
            nodes       => [ qw(localhost:9200) ],
            max_workers => 1,
            proto       => 'http',
            servername  => 'localhost',
            serverport  => 9200,
            username    => ' ',
            password    => ' ',
        },
    }, 

We have some bureaucratic hurdles here in releasing frequent updates to github. I'm working on resolving, but until then. anyone reading this string will have to make these updates by hand. I hope to be able to release updates next week.

[toddbruner]

Matt,

Feel free to reopen if this is still unresolved.