This repository has been archived by the owner on May 23, 2019. It is now read-only.
Troubleshooting CIF
What can I do to if my CIF server isn't working as I expect?
- Reboot the CIF server
- Run the cif ping command with debug
$ cif -p -d
[2016-01-13T03:32:38,391Z][INFO][main:261]: starting up client...
[2016-01-13T03:32:38,392Z][INFO][main:272]: pinging: https://localhost...
[2016-01-13T03:32:38,392Z][DEBUG][CIF::SDK::Client:203]: generating ping...
[2016-01-13T03:32:38,392Z][DEBUG][CIF::SDK::Client:165]: uri created: https://localhost/ping?
[2016-01-13T03:32:38,392Z][DEBUG][CIF::SDK::Client:166]: making request...
[2016-01-13T03:32:38,877Z][INFO][CIF::SDK::Client:170]: status: 200
[2016-01-13T03:32:38,877Z][DEBUG][CIF::SDK::Client:173]: decoding content..
roundtrip: 0.485375 ms
...
[2016-01-13T03:32:44,223Z][INFO][main:393]: done...- Make a cif query with debug
$ cif -q example.com -d
[2016-01-13T02:58:21,076Z][INFO][main:261]: starting up client...
[2016-01-13T02:58:21,076Z][INFO][main:296]: running search...
[2016-01-13T02:58:21,076Z][DEBUG][CIF::SDK::Client:165]: uri created: https://localhost/observables?observable=example.com
[2016-01-13T02:58:21,076Z][DEBUG][CIF::SDK::Client:166]: making request...
[2016-01-13T02:58:21,745Z][INFO][CIF::SDK::Client:170]: status: 200
[2016-01-13T02:58:21,745Z][DEBUG][CIF::SDK::Client:173]: decoding content..
[2016-01-13T02:58:21,745Z][INFO][main:356]: search returned, formatting..
tlp |group |reporttime |observable |cc|asn|confidence|tags |description|rdata|provider |altid_tlp|altid
amber|everyone|2015-12-21T20:01:16Z|example.com| | |25 |search| | |root@localhost| |
amber|everyone|2015-12-21T20:01:18Z|example.com| | |25 |search| | |root@localhost| |
...
[2016-01-13T02:58:21,757Z][INFO][main:393]: done...- Read through all the CIF logs:
$ tail /var/log/cif-router.log
[2016-01-13T03:00:48,136Z][12139][INFO]: staring up..
[2016-01-13T03:00:48,258Z][12141][INFO]: started, waiting for messages..
$ tail /var/log/cif-smrt.log
[2016-01-13T03:00:52,979Z][12325][INFO]: staring up...
[2016-01-13T03:00:52,996Z][12329][INFO]: delaying start for: 4min then running every 60min there after...
[2016-01-13T03:00:52,997Z][12329][INFO]: to run immediately, set: --randomstart 0 or --testmode
[2016-01-13T03:00:52,997Z][12329][INFO]: to see the list of options, use -h
$ tail /var/log/cif-starman.log
[2016-01-13T03:00:52,233Z][12295][INFO]: starting CIF::REST
[2016-01-13T03:00:52,238Z][12297][INFO]: starting CIF::REST
[2016-01-13T03:00:52,255Z][12299][INFO]: starting CIF::REST
$ tail /var/log/cif-worker.log
[2016-01-13T03:00:50,256Z][12188][INFO]: sending ping...
[2016-01-13T03:00:50,313Z][12195][INFO]: staring worker..
[2016-01-13T03:00:50,315Z][12196][INFO]: staring worker..
...
[2016-01-13T03:00:50,337Z][12192][INFO]: starting...- Verify apache is working
$ curl -ik https://localhost/
HTTP/1.1 200 OK
Date: Wed, 13 Jan 2016 13:05:53 GMT
Server: Apache
Vary: Accept-Encoding
Content-Length: 671
Content-Type: text/html;charset=UTF-8
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
...- Read through the Apache logs
$ sudo tail /var/log/apache2/error.log
$ sudo tail /var/log/apache2/ssl_access.log- Verify ElasticSearch is working
$ curl -i 'http://localhost:9200/_cluster/health?pretty'
HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 283
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 155,
"active_shards" : 155,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 155
}- Read through the ElasticSearch logs
$ tail /var/log/elasticsearch/elasticsearch.log- restart cif to look for errors
$ sudo service cif-services restart
* Stopping cif-router [ OK ]
* Starting cif-router [ OK ]
* Stopping cif-worker [ OK ]
* Starting cif-worker [ OK ]
* Stopping cif-starman [ OK ]
* Starting cif-starman [ OK ]
* Stopping cif-smrt [ OK ]
* Starting cif-smrt [ OK ] - Verify the host has enough free disk space
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 7.9G 4.0K 7.9G 1% /dev
tmpfs 1.6G 444K 1.6G 1% /run
/dev/mapper/cifv2--rc6--vg-root 230G 14G 205G 7% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 7.9G 0 7.9G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/sda1 236M 69M 155M 31% /boot- Verify the host has enough free memory
$ free -m
total used free shared buffers cached
Mem: 16047 12496 3551 0 158 2947
-/+ buffers/cache: 9389 6657
Swap: 16383 62 16321- Add '-d' to
CIF_DEBUGGINGin/etc/default/cif
$ cat /etc/default/cif
# Directory where the binary distribution resides
CIF_HOME=/opt/cif
PATH=$CIF_HOME/bin:$PATH
if [ -d /opt/cif/lib/perl5 ]; then
export PERL5LIB=/opt/cif/lib/perl5
fi
# Run as this user ID and group ID
CIF_USER=cif
CIF_GROUP=cif
# data directory
DATA_DIR=/var
LOG_DIR=/var/log
# configuration directory
CONF_DIR=/etc/cif
# add -d to turn on debugging
CIF_DEBUGGING="-d"- Restart all CIF services
$ sudo service cif-services restart
* Stopping cif-router [ OK ]
* Starting cif-router [ OK ]
* Stopping cif-worker [ OK ]
* Starting cif-worker [ OK ]
* Stopping cif-starman [ OK ]
* Starting cif-starman [ OK ]
* Stopping cif-smrt [ OK ]
* Starting cif-smrt [ OK ] - Make a cif query with debug
$ cif -q example.com -d
[2016-01-13T02:58:21,076Z][INFO][main:261]: starting up client...
[2016-01-13T02:58:21,076Z][INFO][main:296]: running search...
[2016-01-13T02:58:21,076Z][DEBUG][CIF::SDK::Client:165]: uri created: https://localhost/observables?observable=example.com
[2016-01-13T02:58:21,076Z][DEBUG][CIF::SDK::Client:166]: making request...
[2016-01-13T02:58:21,745Z][INFO][CIF::SDK::Client:170]: status: 200
[2016-01-13T02:58:21,745Z][DEBUG][CIF::SDK::Client:173]: decoding content..
[2016-01-13T02:58:21,745Z][INFO][main:356]: search returned, formatting..
tlp |group |reporttime |observable |cc|asn|confidence|tags |description|rdata|provider |altid_tlp|altid
amber|everyone|2015-12-21T20:01:16Z|example.com| | |25 |search| | |root@localhost| |
amber|everyone|2015-12-21T20:01:18Z|example.com| | |25 |search| | |root@localhost| | - Read through the apache logs
$ sudo tail /var/log/apache2/ssl_access.log
::1 - - [13/Jan/2016:03:38:23 -1000] "GET /observables?observable=example.com HTTP/1.1" 200 5685 "-" "cif-sdk-perl/2.00_30"
$ sudo tail /var/log/apache2/ssl_access.log- Read through the CIF logs
$ tail /var/log/cif-router.log
$ tail /var/log/cif-smrt.log
$ tail /var/log/cif-starman.log
$ tail /var/log/cif-worker.log- Once done troubleshooting, be sure to turn off CIF debugging and restart all the CIF services; the logging is verbose and will use up a lot of disk space.
