No disk space due to excessive correlations and large tmp misp-dashboard files

[faustus25]

Encounter MISP blackhole requests due to insufficient disk space and the only way to remedy it is to
truncate correlation in mysql to free up disk space:

Details below:

 tail -f /var/www/MISP/app/tmp/logs/error.log
#7 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke(Object(UsersController), Object(CakeRequest))
#8 /var/www/MISP/app/webroot/index.php(92): Dispatcher->dispatch(Object(CakeRequest), Object(CakeResponse))
#9 {main}
Warning: Warning (2): session_write_close(): write failed: No space left on device (28) in [Unknown, line 0]
Trace:
ErrorHandler::handleError() - APP/Lib/cakephp/lib/Cake/Error/ErrorHandler.php, line 230
session_write_close - [internal], line ??
[main] - [internal], line ??

 Warning: Warning (2): session_write_close() [<a href='http://php.net/function.session-write-close'>function.session-write-close</a>]: Failed to write session data (files). Please veri

Further details:

/usr/local/src/misp-dashboard/data/temp-5530.rdb: 2.2G
/usr/local/src/misp-dashboard/data/temp-1404.rdb: 455M
/usr/local/src/misp-dashboard/data/temp-1286.rdb: 326M
/usr/local/src/misp-dashboard/data/temp-1612.rdb: 549M
/usr/local/src/misp-dashboard/data/temp-1582.rdb: 439M
/usr/local/src/misp-dashboard/data/temp-9849.rdb: 2.0G
/usr/local/src/misp-dashboard/data/temp-2020.rdb: 329M
/usr/local/src/misp-dashboard/data/temp-1405.rdb: 407M
/usr/local/src/misp-dashboard/data/dump.rdb: 2.1G
/usr/local/src/misp-dashboard/data/temp-1415.rdb: 436M
/usr/local/src/misp-dashboard/data/temp-1412.rdb: 350M
/usr/local/src/misp-dashboard/data/temp-1435.rdb: 420M
/var/log/lastlog: 264M
/var/log/syslog.1: 107M
/var/lib/redis/dump.rdb: 144M
/var/lib/mysql/misp/logs.ibd: 4.3G
/var/lib/mysql/misp/attributes.ibd: 1.2G
**/var/lib/mysql/misp/correlations.ibd: 72G**
/var/lib/mysql/ibdata1: 268M
/var/www/misp-dashboard/data/dump.rdb: 576M
/var/www/MISP/app/tmp/logs/resque-worker-error.log: 126M

Logging into MYSQL to delete all correlations resolves until the correlations build up again

Resque error log:

main.ERROR: {"queue":"default","id":"a7780ba1ac4f3faeca7604712f6e3511","class":"ServerShell","args":[{"0":"enqueuePull","1":1544719200,"2":"2","3":"2","s_time":1544697602}]} failed: SQLSTATE[HY000]: General error: 1114 The table 'correlations' is full {"type":"fail","log":"SQLSTATE[HY000]: General error: 1114 The table 'correlations' is full","job_id":"a7780ba1ac4f3faeca7604712f6e3511","time":2132390,"worker":"server:6778"} []

[iglocska]

Sounds like you have probably ingested the same feed over and over into new events. To resolve it:

1. Truncate the correlations table
2. modify your csv/freetext feed settings to always use fixed events, not "new event each pull"
3. Delete all the duplicate events coming from the feeds (they will have the same info fields, one per feed)
4. recorrelate the database (server settings -> diagnostics -> legacy tools -> recorrelate attributes)

This should get you up and running again. Let us know if it doesn't resolve it.

[faustus25]

Truncating the correlations table works but I need to delete the high volume attribute Fixed Events for certain events.

Part of the problem is the fact, that Warning List attributes are duplicated in multiple events i.e. "8.8.8.8" and should be ideally mass deleted. The second is that the MISP ZMQ filters through each single attribute for those high volume attribute Fixed Events which is time consuming and eventually the ZMQ listener stops.

[faustus25]

I have remove feeds with excessive attributes but ideally an option to mass delete duplicate false positive attributes would help.

Disk space for misp-dashboard:

/usr/local/src/misp-dashboard/data ------------------------------------------------------------------------------------------------------------------------------------------
                         /..
    2.1 GiB [##########]  temp-5530.rdb
    2.1 GiB [######### ]  dump.rdb
    1.9 GiB [######### ]  temp-9849.rdb
  548.4 MiB [##        ]  temp-1612.rdb
  454.7 MiB [##        ]  temp-1404.rdb
  442.2 MiB [##        ]  temp-1339.rdb
  438.6 MiB [##        ]  temp-1582.rdb
  438.4 MiB [##        ]  temp-1326.rdb
  436.1 MiB [##        ]  temp-1455.rdb
  435.4 MiB [##        ]  temp-1415.rdb
  419.2 MiB [#         ]  temp-1435.rdb
  406.2 MiB [#         ]  temp-1405.rdb
  406.2 MiB [#         ]  temp-1492.rdb
  374.5 MiB [#         ]  temp-1342.rdb
  349.9 MiB [#         ]  temp-1412.rdb
  328.7 MiB [#         ]  temp-2020.rdb
  325.5 MiB [#         ]  temp-1286.rdb
   63.2 MiB [          ] /GeoLite2-City_20180807
   20.0 KiB [          ]  country_code_lat_long.json
@   0.0   B [          ]  GeoLite2-City

Is it recommended to delete all the temp.rdb files?

Also how do you get the ZMQ listener to move onto the next event published? The dashboard is stuck on one event that has an excessive amount of attributes and can't progress to the next one.

[mokaddem]

Indeed, if you publish massive events to the ZMQ channel, it will generate a lot of data.
As of now, you can't skip event that are being processed.