Skip to content
This repository has been archived by the owner on Apr 3, 2023. It is now read-only.

Malice issue ubuntu 16.04 #79

Closed
rufftruffles opened this issue Nov 23, 2018 · 145 comments
Closed

Malice issue ubuntu 16.04 #79

rufftruffles opened this issue Nov 23, 2018 · 145 comments

Comments

@rufftruffles
Copy link

malice@malice:$ go version
go version go1.11.2 linux/amd64
malice@malice:
$ malice scan eicar.pdf
ERRO[0000] database is NOT running, starting now...
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty.readTermInfo(0xc000020960, 0x1e, 0x0, 0x0, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty/gotty.go:232 +0xcdf
github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty.OpenTermInfo(0xc000020065, 0xe, 0xc000020065, 0xe, 0xc000379790)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty/gotty.go:45 +0x287
github.com/maliceio/malice/vendor/github.com/docker/docker/pkg/jsonmessage.DisplayJSONMessagesStream(0xbbc740, 0xc00005e580, 0xbbc940, 0xc000092008, 0x1, 0x1, 0x0, 0x0, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/docker/docker/pkg/jsonmessage/jsonmessage.go:225 +0x55c
github.com/maliceio/malice/malice/docker/client/image.Pull(0xc00001c720, 0xc00028b660, 0x18, 0xb0ef8b, 0x6)
/Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/image/image.go:43 +0x181
github.com/maliceio/malice/malice/docker/client/container.checkContainerRequirements(0xc00001c720, 0xc00028b620, 0xe, 0xc00028b660, 0x18, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/container/utils.go:189 +0x92a
github.com/maliceio/malice/malice/docker/client/container.Start(0xc00001c720, 0x0, 0x0, 0x0, 0xc00028b620, 0xe, 0xc00028b660, 0x18, 0x0, 0xc0003d8040, ...)
/Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/container/start.go:36 +0xf9
github.com/maliceio/malice/malice/database.Start(0xc00001c720, 0x0, 0x0, 0x0, 0x0, 0xc00028b6a0, 0x15, 0x0, 0x0, 0x0, ...)
/Users/blacktop/go/src/github.com/maliceio/malice/malice/database/database.go:45 +0x273
github.com/maliceio/malice/commands.cmdScan(0x7ffc360d46ce, 0x9, 0x0, 0xc00022b600, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/commands/scan.go:62 +0xe26
github.com/maliceio/malice/commands.glob..func1(0xc0000aac60, 0x0, 0xc0000aac60)
/Users/blacktop/go/src/github.com/maliceio/malice/commands/commands.go:25 +0x88
github.com/maliceio/malice/vendor/github.com/urfave/cli.HandleAction(0xa10320, 0xb37b88, 0xc0000aac60, 0xc00022b600, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:490 +0xc8
github.com/maliceio/malice/vendor/github.com/urfave/cli.Command.Run(0xb0e2b0, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb1188e, 0xb, 0x0, ...)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/command.go:210 +0x990
github.com/maliceio/malice/vendor/github.com/urfave/cli.(*App).Run(0xc0000a5380, 0xc000086060, 0x3, 0x3, 0x0, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:255 +0x687
main.main()
/Users/blacktop/go/src/github.com/maliceio/malice/main.go:88 +0x4b2

I tried with multiple GO versions but nothing, any ideas?

Regards,

@blacktop
Copy link
Member

does the same thing happen if you use the deb package? https://github.com/maliceio/malice/releases/tag/v0.3.24

@rufftruffles
Copy link
Author

Yep, just installed it, below is the log:

malice@malice:~$ malice -D scan eicar.pdf
DEBU[0000] Malice config loaded from: /home/malice/.malice/config/config.toml
DEBU[0000] config.toml version: v0.3.24, malice version: 0.3.24
DEBU[0000] Malice plugins loaded from: /home/malice/.malice/plugins/plugins.toml
DEBU[0000] Using 4 PROCS
DEBU[0000] Malice Version: 0.3.24, commit 4a80057, built at 2018-09-09T20:28:58Z
DEBU[0000] Running inside Docker...
DEBU[0000] Connected to docker daemon client ip=localhost port=2375
DEBU[0000] Searching for container: malice-elastic env=development
DEBU[0000] name: malice-elastic container.Name: kibana
DEBU[0000] MATCH: false
DEBU[0000] name: malice-elastic container.Name: elastic
DEBU[0000] MATCH: false
DEBU[0000] Container NOT Found: malice-elastic env=development
ERRO[0000] database is NOT running, starting now...
DEBU[0000] Searching for Network: malice env=development
DEBU[0000] Network FOUND: malice env=development
DEBU[0000] Searching for volume: malice env=development
DEBU[0000] Volume FOUND: malice env=development
DEBU[0000] Volume malice found.
DEBU[0000] Searching for container: malice-elastic env=development
DEBU[0000] name: malice-elastic container.Name: kibana
DEBU[0000] MATCH: false
DEBU[0000] name: malice-elastic container.Name: elastic
DEBU[0000] MATCH: false
DEBU[0000] Container NOT Found: malice-elastic env=development
DEBU[0000] Searching for image: malice/elasticsearch:6.4 env=development
DEBU[0000] Image NOT Found: malice/elasticsearch:6.4 env=development
DEBU[0000] Pulling Image malice/elasticsearch:6.4 env=development exisits=false
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty.readTermInfo(0xc0003a12e0, 0x1e, 0x0, 0x0, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty/gotty.go:232 +0xcdf
github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty.OpenTermInfo(0xc000020065, 0xe, 0xc000020065, 0xe, 0xc000054a30)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty/gotty.go:45 +0x287
github.com/maliceio/malice/vendor/github.com/docker/docker/pkg/jsonmessage.DisplayJSONMessagesStream(0xbbc740, 0xc000396b40, 0xbbc940, 0xc000090008, 0x1, 0xc0001ef501, 0x0, 0x0, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/docker/docker/pkg/jsonmessage/jsonmessage.go:225 +0x55c
github.com/maliceio/malice/malice/docker/client/image.Pull(0xc0003aa720, 0xc000289660, 0x18, 0xb0ef8b, 0x6)
/Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/image/image.go:43 +0x181
github.com/maliceio/malice/malice/docker/client/container.checkContainerRequirements(0xc0003aa720, 0xc000289620, 0xe, 0xc000289660, 0x18, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/container/utils.go:189 +0x92a
github.com/maliceio/malice/malice/docker/client/container.Start(0xc0003aa720, 0x0, 0x0, 0x0, 0xc000289620, 0xe, 0xc000289660, 0x18, 0x0, 0xc00039a780, ...)
/Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/container/start.go:36 +0xf9
github.com/maliceio/malice/malice/database.Start(0xc0003aa720, 0x0, 0x0, 0x0, 0x0, 0xc0002896a0, 0x15, 0x0, 0x0, 0x0, ...)
/Users/blacktop/go/src/github.com/maliceio/malice/malice/database/database.go:45 +0x273
github.com/maliceio/malice/commands.cmdScan(0x7ffcc22376ce, 0x9, 0x0, 0xc000229600, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/commands/scan.go:62 +0xe26
github.com/maliceio/malice/commands.glob..func1(0xc0000a6c60, 0x0, 0xc0000a6c60)
/Users/blacktop/go/src/github.com/maliceio/malice/commands/commands.go:25 +0x88
github.com/maliceio/malice/vendor/github.com/urfave/cli.HandleAction(0xa10320, 0xb37b88, 0xc0000a6c60, 0xc000229600, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:490 +0xc8
github.com/maliceio/malice/vendor/github.com/urfave/cli.Command.Run(0xb0e2b0, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb1188e, 0xb, 0x0, ...)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/command.go:210 +0x990
github.com/maliceio/malice/vendor/github.com/urfave/cli.(*App).Run(0xc0000a3380, 0xc000094000, 0x4, 0x4, 0x0, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:255 +0x687
main.main()
/Users/blacktop/go/src/github.com/maliceio/malice/main.go:88 +0x4b2

@rufftruffles
Copy link
Author

docker :

malice@malice:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
eccf14ba5fd0 malice/kibana "/entrypoint.sh kiban" 16 minutes ago Up 16 minutes 0.0.0.0:5601->5601/tcp kibana
9bf7175ae809 malice/elasticsearch "/elastic-entrypoint." 27 minutes ago Up 27 minutes 0.0.0.0:9200->9200/tcp, 9300/tcp elastic

@blacktop
Copy link
Member

other people have said the first scan fails because it also tries to create/start the database, but that subsequent scans do work, because the db is already running?

@rufftruffles
Copy link
Author

When trying to install/update plugins:

malice@malice:~$ malice -D plugin update --all
DEBU[0000] Malice config loaded from: /home/malice/.malice/config/config.toml
DEBU[0000] config.toml version: v0.3.24, malice version: 0.3.24
DEBU[0000] Malice plugins loaded from: /home/malice/.malice/plugins/plugins.toml
DEBU[0000] Using 4 PROCS
DEBU[0000] Malice Version: 0.3.24, commit 4a80057, built at 2018-09-09T20:28:58Z
DEBU[0000] Running inside Docker...
DEBU[0000] Connected to docker daemon client ip=localhost port=2375
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty.readTermInfo(0xc000020300, 0x1e, 0x0, 0x0, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty/gotty.go:232 +0xcdf
github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty.OpenTermInfo(0xc000020065, 0xe, 0xc000020065, 0xe, 0xc0003ce1e0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/Nvveen/Gotty/gotty.go:45 +0x287
github.com/maliceio/malice/vendor/github.com/docker/docker/pkg/jsonmessage.DisplayJSONMessagesStream(0xbbc740, 0xc00005e140, 0xbbc940, 0xc000092008, 0x1, 0x1, 0x0, 0x0, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/docker/docker/pkg/jsonmessage/jsonmessage.go:225 +0x55c
github.com/maliceio/malice/malice/docker/client/image.Pull(0xc0003a8720, 0xb0f593, 0x7, 0xb0ef8b, 0x6)
/Users/blacktop/go/src/github.com/maliceio/malice/malice/docker/client/image/image.go:43 +0x181
github.com/maliceio/malice/plugins.UpdateEnabledPlugins(0xc0003a8720)
/Users/blacktop/go/src/github.com/maliceio/malice/plugins/plugins.go:248 +0x8f
github.com/maliceio/malice/commands.cmdUpdatePlugin(0x0, 0x0, 0x1, 0x0, 0xc00022b6e0)
/Users/blacktop/go/src/github.com/maliceio/malice/commands/plugin.go:161 +0x20f
github.com/maliceio/malice/commands.glob..func8(0xc0000aaf20, 0x0, 0xc0000aaf20)
/Users/blacktop/go/src/github.com/maliceio/malice/commands/commands.go:138 +0xc1
github.com/maliceio/malice/vendor/github.com/urfave/cli.HandleAction(0xa10320, 0xb37bc0, 0xc0000aaf20, 0xc00022b600, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:490 +0xc8
github.com/maliceio/malice/vendor/github.com/urfave/cli.Command.Run(0xb0f147, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb12fee, 0xd, 0x0, ...)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/command.go:210 +0x990
github.com/maliceio/malice/vendor/github.com/urfave/cli.(*App).RunAsSubcommand(0xc0000a5520, 0xc0000aac60, 0x0, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:379 +0x7ef
github.com/maliceio/malice/vendor/github.com/urfave/cli.Command.startApp(0xb0effd, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb1ed92, 0x1f, 0x0, ...)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/command.go:298 +0x808
github.com/maliceio/malice/vendor/github.com/urfave/cli.Command.Run(0xb0effd, 0x6, 0x0, 0x0, 0x0, 0x0, 0x0, 0xb1ed92, 0x1f, 0x0, ...)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/command.go:98 +0x1237
github.com/maliceio/malice/vendor/github.com/urfave/cli.(*App).Run(0xc0000a5380, 0xc000096000, 0x5, 0x5, 0x0, 0x0)
/Users/blacktop/go/src/github.com/maliceio/malice/vendor/github.com/urfave/cli/app.go:255 +0x687
main.main()
/Users/blacktop/go/src/github.com/maliceio/malice/main.go:88 +0x4b2

@blacktop
Copy link
Member

I think it was a bad design decision on my part to have malware create the DBs 🤔

@rufftruffles
Copy link
Author

I think it was a bad design decision on my part to have malware create the DBs

haha, what should I do in this scenario now?

@blacktop
Copy link
Member

can you tell me more about your host? I ran this on an AWS node when I filled the demo.malice.io instance will all of its data

@blacktop
Copy link
Member

I used debian package and it worked perfectly :(

@rufftruffles
Copy link
Author

can you tell me more about your host? I ran this on an AWS node when I filled the demo.malice.io instance will all of its data

It's running ubuntu 16.04 on a KVM based vm, installed docker-engine, followed te following doc to install go & malice using source https://github.com/maliceio/malice/blob/master/docs/installation/linux/install.md

then also started elastic search container.

@rufftruffles
Copy link
Author

I can give you access to the vm if you need to take a look and issue a fix for future releases?

@blacktop
Copy link
Member

did you do this step:

https://github.com/maliceio/malice#known-issues-warning

sudo sysctl -w vm.max_map_count=262144

@rufftruffles
Copy link
Author

did you do this step:

https://github.com/maliceio/malice#known-issues-warning

sudo sysctl -w vm.max_map_count=262144

Yes, as soon as I setup the vm lol

@blacktop
Copy link
Member

Another thing you can do is get elasticsearch running outside of docker/malice, then you can point malice to it with the env var MALICE_ELASTICSEARCH_URL=<host>:<port>

@rufftruffles
Copy link
Author

Another thing you can do is get elasticsearch running outside of docker/malice, then you can point malice to it with the env var MALICE_ELASTICSEARCH_URL=<host>:<port>

I don't think that's gonna work, should I send you the vm credentials via email to take a look in your free time? I'm sure this will help a lot of people out there struggling with the same issue.

@blacktop
Copy link
Member

I'm going to spin up a vagrantbox for xenial and check real quick

@rufftruffles
Copy link
Author

I'm going to spin up a vagrantbox for xenial and check real quick

Perfecto!

@blacktop
Copy link
Member

blacktop commented Nov 24, 2018

Ok so I have a solution... but you are NOT going to like it! 😬

It's a friggin BUG in one of the docker src code dependancies! :rage4:

I was able to recreate your issue and the fix for me was:

$ TERM="" malice plugin update clamav

@blacktop
Copy link
Member

It looks like docker-it-self had the SAME issue and they solved a while back and it fixed it by overriding the dep with another repo since the repo is dead. I copied their solution and cut another release. Can you please test and let me know.

That was a VERY embarrassing bug, thank you for pointing that out to me!

@rufftruffles
Copy link
Author

rufftruffles commented Nov 24, 2018

It looks like docker-it-self had the SAME issue and they solved a while back and it fixed it by overriding the dep with another repo since the repo is dead. I copied their solution and cut another release. Can you please test and let me know.

That was a VERY embarrassing bug, thank you for pointing that out to me!

Hey there! Your fix worked wonders, everything is fixed except for the elasticsearch: here are a few outputs:

alice@malice:~$ malice scan f.pdf
#### File
| Field  | Value                                                            |
| ------ | ---------------------------------------------------------------- |
| Name   | f.pdf                                                            |
| Path   | f.pdf                                                            |
| Size   | 2.061kB                                                          |
| MD5    | 911dd1610034027a924387d42f56bdf0                                 |
| SHA1   | 6ce8d59428b6a646ac5eb440b540e8984ece5b08                         |
| SHA256 | 4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 |
FATA[0001] scan cmd failed to store file info: failed to connect to database: failed to ping elasticsearch: Get http://localhost:9200/: dial tcp [::1]:9200: connect: connection refused 

The container was created when I ran malice scan first time:

malice@malice:~$ malice scan eicar.pdf
ERRO[0000] database is NOT running, starting now...     
ERRO[0000] Network malice does not exist, creating now...  env=development exisits=false network=malice
INFO[0000] Created Network: malice                       env=development name=malice
INFO[0000] Created Volume: malice                        env=development
6.5: Pulling from malice/elasticsearch
4fe2ade4980c: Pull complete 
c9dbc0055e45: Pull complete 
d4511882860e: Pull complete 
2772c7b6d4e2: Pull complete 
589015d5f852: Pull complete 
e1dae11492e9: Pull complete 
9ecd75eb0b8e: Pull complete 
0f42f265a9ba: Pull complete 
Digest: sha256:0fdbffc5b93cb612bf4d64c93b8627a6438d293a3b0394e0f4054545f99500b8
Status: Downloaded newer image for malice/elasticsearch:6.5
INFO[0012] elasticsearch container started               assigned_ip=172.17.0.2 docker_ip=localhost name=/malice-elastic port="[9200]" runtime_env=development
FATA[0032] failed to start to database: connecting to elasticsearch timed out after 20 seconds: failed to ping elasticsearch: Get http://localhost:9200/: dial tcp [::1]:9200: connect: connection refused
malice@malice:~$ docker start 9ae59e8f4012
9ae59e8f4012
malice@malice:~$ malice scan f.pdf
#### File
| Field  | Value                                                            |
| ------ | ---------------------------------------------------------------- |
| Name   | f.pdf                                                            |
| Path   | f.pdf                                                            |
| Size   | 2.061kB                                                          |
| MD5    | 911dd1610034027a924387d42f56bdf0                                 |
| SHA1   | 6ce8d59428b6a646ac5eb440b540e8984ece5b08                         |
| SHA256 | 4daf4edbb04383b93094e89636f303bb11ab687636a4d40813e930c213c3b513 |
**FATA[0002] scan cmd failed to store file info: failed to connect to database: failed to ping elasticsearch: Get http://localhost:9200/: read tcp [::1]:46522->[::1]:9200: read: connection reset by peer** 

@blacktop
Copy link
Member

can you check the elastic logs. please see #80

@rufftruffles
Copy link
Author

rufftruffles commented Nov 24, 2018

Next steps: removed elastic container and redeployed manually:

malice@malice:~$ docker rm 9ae59e8f4012
9ae59e8f4012
malice@malice:~$ docker run -d --name elastic -p 9200:9200 malice/elasticsearch
Unable to find image 'malice/elasticsearch:latest' locally
latest: Pulling from malice/elasticsearch
4fe2ade4980c: Already exists 
c9dbc0055e45: Already exists 
d4511882860e: Already exists 
2772c7b6d4e2: Already exists 
589015d5f852: Already exists 
e1dae11492e9: Already exists 
9ecd75eb0b8e: Already exists 
0f42f265a9ba: Already exists 
Digest: sha256:c7dbed8f3054499e2d11991cab4aef641ba5a63b38874e9372915473a5ef5252
Status: Downloaded newer image for malice/elasticsearch:latest
241c1addf6be974697d1c14096de073d36f122f2561c4c5100bc571da2d8af27
malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                              NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   7 seconds ago       Up 5 seconds        0.0.0.0:9200->9200/tcp, 9300/tcp   elastic
malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                              NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   12 seconds ago      Up 10 seconds       0.0.0.0:9200->9200/tcp, 9300/tcp   elastic
malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                              NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   13 seconds ago      Up 12 seconds       0.0.0.0:9200->9200/tcp, 9300/tcp   elastic
malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                              NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   15 seconds ago      Up 13 seconds       0.0.0.0:9200->9200/tcp, 9300/tcp   elastic
malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                              NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   15 seconds ago      Up 14 seconds       0.0.0.0:9200->9200/tcp, 9300/tcp   elastic
malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                       PORTS               NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   38 seconds ago      Exited (78) 18 seconds ago                       elastic

@blacktop
Copy link
Member

It might also be that malice isn't giving elasticsearch enough time to start on your machine

@rufftruffles
Copy link
Author

can you check the elastic logs. please see #80

There you go:

malice@malice:~$ docker logs -f elastic
[2018-11-24T14:35:00,648][WARN ][o.e.c.l.LogConfigurator  ] [unknown] Some logging configurations have %marker but don't have %node_name. We will automatically add %node_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
  /usr/share/elasticsearch/config/log4j2.properties
[2018-11-24T14:35:01,262][INFO ][o.e.e.NodeEnvironment    ] [YiAkBn1] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/vda1)]], net usable_space [15.1gb], net total_space [28.5gb], types [ext3]
[2018-11-24T14:35:01,263][INFO ][o.e.e.NodeEnvironment    ] [YiAkBn1] heap size [990.7mb], compressed ordinary object pointers [true]
[2018-11-24T14:35:01,269][INFO ][o.e.n.Node               ] [YiAkBn1] node name derived from node ID [YiAkBn1LScK9-toK0v9DBw]; set [node.name] to override
[2018-11-24T14:35:01,270][INFO ][o.e.n.Node               ] [YiAkBn1] version[6.5.0], pid[1], build[oss/tar/816e6f6/2018-11-09T18:58:36.352602Z], OS[Linux/4.4.0-109-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
[2018-11-24T14:35:01,270][INFO ][o.e.n.Node               ] [YiAkBn1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/usr/share/elasticsearch/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar]
[2018-11-24T14:35:02,703][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [aggs-matrix-stats]
[2018-11-24T14:35:02,703][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [analysis-common]
[2018-11-24T14:35:02,703][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [ingest-common]
[2018-11-24T14:35:02,703][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [lang-expression]
[2018-11-24T14:35:02,704][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [lang-mustache]
[2018-11-24T14:35:02,704][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [lang-painless]
[2018-11-24T14:35:02,704][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [mapper-extras]
[2018-11-24T14:35:02,704][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [parent-join]
[2018-11-24T14:35:02,704][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [percolator]
[2018-11-24T14:35:02,705][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [rank-eval]
[2018-11-24T14:35:02,705][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [reindex]
[2018-11-24T14:35:02,705][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [repository-url]
[2018-11-24T14:35:02,705][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [transport-netty4]
[2018-11-24T14:35:02,705][INFO ][o.e.p.PluginsService     ] [YiAkBn1] loaded module [tribe]
[2018-11-24T14:35:02,706][INFO ][o.e.p.PluginsService     ] [YiAkBn1] no plugins loaded
[2018-11-24T14:35:08,150][INFO ][o.e.d.DiscoveryModule    ] [YiAkBn1] using discovery type [zen] and host providers [settings]
[2018-11-24T14:35:08,920][INFO ][o.e.n.Node               ] [YiAkBn1] initialized
[2018-11-24T14:35:08,921][INFO ][o.e.n.Node               ] [YiAkBn1] starting ...
[2018-11-24T14:35:09,168][INFO ][o.e.t.TransportService   ] [YiAkBn1] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
[2018-11-24T14:35:09,187][INFO ][o.e.b.BootstrapChecks    ] [YiAkBn1] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2018-11-24T14:35:09,202][INFO ][o.e.n.Node               ] [YiAkBn1] stopping ...
[2018-11-24T14:35:09,346][INFO ][o.e.n.Node               ] [YiAkBn1] stopped
[2018-11-24T14:35:09,347][INFO ][o.e.n.Node               ] [YiAkBn1] closing ...
[2018-11-24T14:35:09,370][INFO ][o.e.n.Node               ] [YiAkBn1] closed

@blacktop
Copy link
Member

whoa I don't understand how you can have so many elastics running at the same time, docker should have complained that you already had something listening on port 9200 ??

@blacktop
Copy link
Member

so that looks like it needs your to run

echo "vm.max_map_count=262144" | sudo tee -a /etc/sysctl.conf
sudo sysctl -w vm.max_map_count=262144

I assume if you DID run sudo sysctl -w vm.max_map_count=262144 then you rebooted the vm? because it doesn't presist unless you write it to /etc/sysctl.conf

@rufftruffles
Copy link
Author

whoa I don't understand how you can have so many elastics running at the same time, docker should have complained that you already had something listening on port 9200 ??

Oh no, take a look again, I kept running ps -a to see when the container dies, check the result of last ps -a (container died after 38 secs of startup):

malice@malice:~$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                       PORTS               NAMES
241c1addf6be        malice/elasticsearch   "/elastic-entrypoint."   38 seconds ago      Exited (78) 18 seconds ago                       elastic

@blacktop
Copy link
Member

ah ok, also I think malice expects the container to be called malice-elastic

@rufftruffles
Copy link
Author

ah ok, also I think malice expects the container to be called malice-elastic

Oh crap! I had reinstalled the vm and forgot to update max map count :D

@blacktop
Copy link
Member

So ya, when the docker logs -f malice-elastic says:

ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

It means you need to run: sudo sysctl -w vm.max_map_count=262144

@rufftruffles
Copy link
Author

Try the vm I sent you, it's not docker

@blacktop
Copy link
Member

I can see it working on your vm at least the webui.

I can update defs in docker but scanning keeps saying Operation not permited?

@blacktop
Copy link
Member

the VM's cli is working great too!

root@Av:~# /opt/kaspersky/kav4fs/bin/kav4fs-control --scan-file eicar.com.txt
Objects scanned:     1
Threats found:       1
Riskware found:      0
Infected:            1
Suspicious:          0
Cured:               0
Moved to quarantine: 0
Removed:             0
Not cured:           0
Scan errors:         0
Password protected:  0
Corrupted:           0

@rufftruffles
Copy link
Author

Did the install go fine on your docker?

@blacktop
Copy link
Member

you should use your awesome google-fu and tell me why this would be happening:

root@21d8c2418077:/# /opt/kaspersky/kav4fs/bin/kav4fs-control --scan-file /malware/EICAR
Couldn't scan file /malware/EICAR
Description: Operation isn't allowed

@blacktop
Copy link
Member

I can install and add a key AND update the definitions, but not scan??

@blacktop
Copy link
Member

Also this:

root@21d8c2418077:/# /opt/kaspersky/kav4fs/bin/kav4fs-control --query-status
Error: couldn't query the status: black list is corrupted or missing.

@rufftruffles
Copy link
Author

add to file /etc/sudoers this string:

username ALL=NOPASSWD: /opt/kaspersky/kav4fs/bin/kav4fs-control

username - name of existing user that you want to use for on demand scan

and then $sudo /opt/kaspersky/kav4fs/bin/kav4fs-control --scan-file /my/path/to/testfile.dump.gz

@blacktop
Copy link
Member

docker is different than a real VM:

root@21d8c2418077:/# cat /etc/sudoers
cat: /etc/sudoers: No such file or directory

but I agree that it might be because I am running it all as root in docker

@blacktop
Copy link
Member

I also know that docker blacklists some types of syscalls and that also might be it?

@rufftruffles
Copy link
Author

@rufftruffles
Copy link
Author

Yes that could be, also Kaspersky says they do not support docker for ka4fs

@blacktop
Copy link
Member

guess who has two thumbs and just got kaspersky to work inside docker.....

@blacktop
Copy link
Member

THIS GUY 👍 😎 👍

@rufftruffles
Copy link
Author

Holy crap, how did you do it? :D

@blacktop
Copy link
Member

locale.... the whole friggin reason NOTHING was working..... locale locale locale

@rufftruffles
Copy link
Author

Wtfff :o

@blacktop
Copy link
Member

I AM THE DOCKER MASTER!!!!!! ✊

@rufftruffles
Copy link
Author

hahahaah no doubt :D

@blacktop
Copy link
Member

https://github.com/malice-plugins/kaspersky/blob/master/README.md#thanks

@rufftruffles
Copy link
Author

@blacktop your next target should be quick heal Linux version: https://www.seqrite.com/seqrite-for-linux

@rufftruffles
Copy link
Author

@blacktop
Copy link
Member

blacktop commented Dec 2, 2018

I opened this issue to get those AVs you suggested and this page takes sooooo long to load. 🤣 Issues aren't supposed to get this big!!

@blacktop
Copy link
Member

blacktop commented Dec 2, 2018

@rufftruffles
Copy link
Author

Haha this issue will go on and on and on until we have 1000 comments in here :D

@absolis
Copy link

absolis commented Feb 21, 2019

[root@***]# docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -e MALICE_VT_API=$MALICE_VT_API malice/engine lookup SOMEHASH
ERRO[0000] database is NOT running, starting now...
INFO[0000] elasticsearch container started assigned_ip=172.17.0.3 docker_ip=localhost name=/malice-elastic port="[9200]" runtime_env=development
FATA[0020] failed to start to database: connecting to elasticsearch timed out after 18 seconds: failed to ping elasticsearch: Get http://localhost:9200/: dial tcp [::1]:9200: connect: connection refused

immediate retry
[root@***]# docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -e MALICE_VT_API=$MALICE_VT_API malice/engine lookup SOMEHASH
FATA[0000] cmd lookup failed to store hash: Database.Plugins is empty (you must set this field to use this function)

HELP PLEACE!!

@blacktop
Copy link
Member

I think you might need to add --network="host" see #84

@absolis
Copy link

absolis commented Feb 25, 2019

I think you might need to add --network="host" see #84

[root@*** ~]# docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -e MALICE_VT_API=$MALICE_VT_API -e ES_JAVA_OPTS="-Xms4g -Xmx4g" --network="host" malice/engine lookup SOMEHASH
ERRO[0000] database is NOT running, starting now...
INFO[0001] elasticsearch container started assigned_ip=172.17.0.3 docker_ip=localhost name=/malice-elastic port="[9200]" runtime_env=development
FATA[0021] failed to start to database: connecting to elasticsearch timed out after 18 seconds: failed to ping elasticsearch: Get http://localhost:9200/: dial tcp [::1]:9200: connect: connection refused

[root@*** ~]# docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -e MALICE_VT_API=$MALICE_VT_API -e ES_JAVA_OPTS="-Xms4g -Xmx4g" --network="host" malice/engine lookup SOMEHASH
FATA[0000] cmd lookup failed to store hash: Database.Plugins is empty (you must set this field to use this function)

[root@*** ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ae450c91c2e1 malice/elasticsearch:6.5 "/elastic-entrypoi..." 2 minutes ago Exited (78) 2 minutes ago malice-elastic

:(

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants