Worker can't be run with process (doesn't have command)

[gru3zi]

Request Type

Bug

Work Environment

Question	Answer
OS version (server): Ubuntu 20.04
Cortex version: 3.1.7-1
Package Type Docker 20.10.21, build baeda1f

Problem Description

I tried to troubleshoot the issue by searching for a similar error but was not able to find someone who also encountered the error "worker 1c6e204a18bdf14698dcf3d4975b72c8 can't be run with process (doesn't have command)"

I also searched the logs for permission denied issues but did not find anything in /var/log/cortex/application.log

Steps to Reproduce

1. Installed Cortex via Docker-Compose (template provided below)
2. Installed analyzers on Ubuntu host in /opt folder via git. Dependencies were also installed using the script "for I in $(find opt/Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip3 install -U -r $I || true; done"
3. Configure analyzers with all the proper keys
4. Running any analyzer results in a error "Worker cannot be run"

Complementary information

log sample output

2022-11-05 18:42:23,886 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-46 - Job gpgZSYQBhvjiDf8Q3XkG has be updated (JsDefined("InProgress"))
2022-11-05 18:42:23,887 [WARN] from org.thp.cortex.services.JobRunnerSrv in application-analyzer-48 - worker 1c6e204a18bdf14698dcf3d4975b72c8 can't be run with process (doesn't have command)
2022-11-05 18:42:24,895 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-47 - Job gpgZSYQBhvjiDf8Q3XkG has be updated (JsDefined("Failure"))
2022-11-05 18:42:24,896 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-46 - Job gpgZSYQBhvjiDf8Q3XkG has finished with status Failure
2022-11-05 18:42:28,741 [INFO] from org.thp.cortex.services.AccessLogFilter in application-akka.actor.default-dispatcher-30 - 192.168.80.194 GET /api/alert took 7ms and returned 200 2 bytes
2022-11-05 18:42:32,949 [INFO] from org.thp.cortex.services.AccessLogFilter in application-akka.actor.default-dispatcher-30 - 192.168.80.194 POST /api/job/_search?range=0-50&sort=-createdAt took 9ms and returned 200

Another error im seeing in the logs

2022-11-06 10:04:11,177 [INFO] from org.thp.cortex.services.AccessLogFilter in application-akka.actor.default-dispatcher-4465 - 192.168.80.194 GET /api/status took 0ms and returned 200 278 bytes
2022-11-06 10:04:11,179 [ERROR] from org.elastic4play.controllers.Authenticated in application-akka.actor.default-dispatcher-4473 - Authentication failure:
session: AuthenticationError User session has expired
pki: AuthenticationError Certificate authentication is not configured
key: AuthenticationError Authentication header not found
init: AuthenticationError Use of initial user is forbidden because users exist in database

screenshots

configuration dumps

Link to docker-compose file and application.conf files

https://github.com/gru3zi/cortexerror/tree/main

[michlschmid]

Thanks for reporting.
I'm expieriencing exactly the same issue in my docker-based proof-of-concept setup for a TH4 -> TH5 migration.
Can't get the analyzers up and running.

@nadouani can we provide some helpful log for diagnosis?

Regards, Michl

[gru3zi]

I ended up getting it working by installing cortex on a separate docker container. I followed this guide

https://docs.thehive-project.org/cortex/installation-and-configuration/

and used the following command on a Ubuntu 20.04 machine.

wget -q -O /tmp/install.sh https://archives.strangebee.com/scripts/install.sh ; sudo -v ; bash /tmp/install.sh

Its not ideal running a separate container as they could all be together in one container but hey it works.

[michlschmid]

Hi @gru3zi,

thanks for your reply. I finally found the root cause of the problem - after hours of search ;-)

The problem is "how" the Cortex container is designed as it relies isself on the Docker process of its host when it comes to start the needed "analyzer containers" (now called "neurons" in the docs).

Therefore you have to exactly share the host paths where the Cortex container may write its jobs temp data, results etc. and where it finds the hosts Docker process' socket.

Thats a bit tricky to findout as it was not that clear in the docs to me at the first sight - but others struggled herein as well in related cases like #360 :-)

As we probably used the same tutorial (suppose: https://ls111.me/how-to-integrate-cortex-misp-with-thehive-in-your-soc/) as a base for our docker-compose.yml - here's what I've fixed to get it up and running:

  cortex:
    image: thehiveproject/cortex:latest
    restart: unless-stopped
    # Memory Limit of this Container within the Docker Ecosystem
    # The value should be slightly above the expected application load values
    mem_limit: 1000m
    environment:
      # IMPORTANT
      # The Cortex Container uses this Docker Host for running its own Analyzer Containers (called "Neurons") on analysis demand.
      # -> These two folders must exist on the Docker host!
      # -> The Container needs access to these two folders!
      - job_directory=/tmp/cortex-jobs
      - docker-job-directory=/var/run/cortex/jobs
    volumes:
      # IMPORTANT
      # The Cortex Container uses this Docker Hosts "Docker Process" for running its own Analyzer Containers (called "Neurons") on analysis demand.
      # -> Configure these settings to match your Docker Host paths (left side of the ":") to the right paths!      
      - /var/run/docker.sock:/var/run/docker.sock
      # -> This folder must exist on the Docker host and the correct Docker Host path (left side of the ":") needs to be specified!
      - /tmp/cortex-jobs:/tmp/cortex-jobs
    depends_on:
      - elasticsearch
    ports:
      - "0.0.0.0:9001:9001"
    networks:
      - SOC_NET

The originally used Docker data volume "cortexdata" can be removed as it's no longer neccesary in this usecase.

Hope that helps.

Cheers,

Michl

[michlschmid]

@nadouani this issue could probably be closed as it's not a bug more a config issue.

Probably you could clearify things in the docs? Maybe with one more of the well done setup illustrations?

Cheers,

Michl

[gru3zi]

You are an absolute star Michl! Thank you for taking the time to troubleshoot and get this working. I can confirm your changes fixed it for me too. Have a nice evening :)

[d1390up]

I have the same problem but I don't have docker. I have Ubuntu 22.04 (64 Bit) and Cortex 3.1.8-1.

Default installation according to the guide: https://docs.strangebee.com/cortex/installation-and-configuration/analyzers-responders/

/etc/cortex/application.conf

Can you help me?