Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: Snorby::Jobs::DailyCacheJob failed with DataObjects::DataError: Data too long for column 'src_ips' at row 1 - 2 failed attempts #109

Closed
PacketShepard opened this issue Aug 9, 2011 · 6 comments

Comments

@PacketShepard
Copy link

Snorby::Jobs::DailyCacheJob failed with DataObjects::DataError: Data too long for column 'src_ips' at row 1 - 0 failed attempts
Snorby::Jobs::SensorCacheJob completed after 25.3841
Snorby::Jobs::DailyCacheJob failed with DataObjects::DataError: Data too long for column 'src_ips' at row 1 - 1 failed attempts
Snorby::Jobs::SensorCacheJob completed after 24.4008
Snorby::Jobs::DailyCacheJob failed with DataObjects::DataError: Data too long for column 'src_ips' at row 1 - 2 failed attempts
PERMANENTLY removing Snorby::Jobs::DailyCacheJob because of 3 consecutive failures.
Snorby::Jobs::SensorCacheJob completed after 7200.1682
6 jobs processed at 0.0005 j/s, 3 failed ...

This is the issue I'm currently seeing in Snorby 2.3.6 on RHEL6. The DB is on a separate server running RHEL5.6 with MySQL 5.

The SensorCacheJob also crashes after this (or at least it no longer appears on the worker page). I've tried everything I can think of. I recreated the database and it worked okay with < 15,000 alerts. I've also rebuilt the cache from stratch and tried updating different modules of the install. I've also tried running straight from rails without passenger to see if that makes a different (I can at least control delayed_jobs through the interface now. You can't start/stop/restart/ in passenger).

Thank you for your assistance. I really like Snorby's dashboard and email summaries when they're working well (and so does my director!).

@terracatta
Copy link
Contributor

Can you log into the DB and post some back of examples of dst and src IPs
from recent alerts?

On Tue, Aug 9, 2011 at 10:19 AM, dmerida <
reply@reply.github.com>wrote:

Snorby::Jobs::DailyCacheJob failed with DataObjects::DataError: Data too
long for column 'src_ips' at row 1 - 0 failed attempts
Snorby::Jobs::SensorCacheJob completed after 25.3841
Snorby::Jobs::DailyCacheJob failed with DataObjects::DataError: Data too
long for column 'src_ips' at row 1 - 1 failed attempts
Snorby::Jobs::SensorCacheJob completed after 24.4008
Snorby::Jobs::DailyCacheJob failed with DataObjects::DataError: Data too
long for column 'src_ips' at row 1 - 2 failed attempts
PERMANENTLY removing Snorby::Jobs::DailyCacheJob because of 3 consecutive
failures.
Snorby::Jobs::SensorCacheJob completed after 7200.1682
6 jobs processed at 0.0005 j/s, 3 failed ...

This is the issue I'm currently seeing in Snorby 2.3.6 on RHEL6. The DB is
on a separate server running RHEL5.6 with MySQL 5.

The SensorCacheJob also crashes after this (or at least it no longer
appears on the worker page). I've tried everything I can think of. I
recreated the database and it worked okay with < 15,000 alerts. I've also
rebuilt the cache from stratch and tried updating different modules of the
install. I've also tried running straight from rails without passenger to
see if that makes a different (I can at least control delayed_jobs through
the interface now. You can't start/stop/restart/ in passenger).

Thank you for your assistance. I really like Snorby's dashboard and email
summaries when they're working well (and so does my director!).

Reply to this email directly or view it on GitHub:
#109

@PacketShepard
Copy link
Author

Here are ten from the bottom of the iphdr table. I grouped them in descending order and then limited by 10. I also converted the fields back into IPs in the two right columns. This CSV format was the best that I could display here.

ip_src,ip_dst,inet_ntoa(iphdr.ip_src),inet_ntoa(iphdr.ip_dst)
2639878987,3091844678,157.89.91.75,184.73.206.70
1612554794,2639864907,96.29.162.42,157.89.36.75
2798456175,2639865047,166.205.13.111,157.89.36.215
2793999750,2639865047,166.137.13.134,157.89.36.215
1151891210,2639868495,68.168.119.10,157.89.50.79
2793999978,2639864944,166.137.14.106,157.89.36.112
2639910492,1482307761,157.89.214.92,88.90.56.177
241245026,2639864941,14.97.27.98,157.89.36.109
2798456578,2639864944,166.205.15.2,157.89.36.112
840426077,2639864940,50.23.226.93,157.89.36.108

The issue seems to stem from multiple ips loading into an object during a cache job. I can see the full error in verbose mode, but it's too great to post here.

@PacketShepard
Copy link
Author

When I say verbose mode, I'm referring to running the cache jobs manually via "rails c."

It's like so:
Snorby::Jobs::SensorCacheJob.new(true).perform
Snorby::Jobs::DailyCacheJob.new(true).perform

The output from the error in this case is many thousands of lines long and therefore too long to post in issues. I could email the full error if necessary.

@PacketShepard
Copy link
Author

Bump

Any other comments on a fix for this terracatta?

@acmarques
Copy link
Contributor

Hi,

I got the same issue today, turns out the reason is that the columns "signature_metrics", "severity_metrics", "dst_ips" and "src_ips" from the tables "caches" and "daily_caches" are from type TEXT, being thus limited to 65,535 bytes. What solved my problem was updating these columns to MEDIUMTEXT, so they support up to ~16MB text.

So, a quick fix is login into your mysql server and run:

alter table daily_caches change signature_metrics signature_metrics mediumtext;
alter table daily_caches change severity_metrics severity_metrics mediumtext;
alter table daily_caches change dst_ips dst_ips mediumtext;
alter table daily_caches change src_ips src_ips mediumtext;

alter table caches change signature_metrics signature_metrics mediumtext;
alter table caches change severity_metrics severity_metrics mediumtext;
alter table caches change dst_ips dst_ips mediumtext;
alter table caches change src_ips src_ips mediumtext;

I'm working on a definitive fix for this, I'll probably push it in the next few days.

@acmarques
Copy link
Contributor

Closing this one, commit 09db614 fixes it.

New Snorby installations will have the fields of type Object mapped automatically to MEDIUMTEXT on mysql.
For existing installations it's necessary to manually update (see last comment) or run rake snorby:hard_reset (beware - all data will be lost with this command)

shadowbq pushed a commit to shadowbq/snorby that referenced this issue Apr 11, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants