docker yeti_1 & yeti-beat_1 crashing BSON field 'no_cursor_timeout' is an unknown field.

[starskizzle]

Description

yeti_1 and yeti-beat_1 issue Errors

yeti_1 OperationError: Update failed (BSON field 'no_cursor_timeout' is an unknown field.)

yeti-beat_1 | mongoengine.errors.OperationError: Update failed (BSON field 'no_cursor_timeout' is an unknown field.)

Environment

Fresh Ubuntu 18.04.3 LTS install
Fresh Docker CE Install (19.03.1)
Fresh Docker-Compose (1.24.1)

Steps to Reproduce

1. https://github.com/yeti-platform/yeti/tree/master/extras/docker
2. git clone https://github.com/yeti-platform/yeti
3. cd yeti/extras/docker/
4. docker-compose -f docker-compose-full.yml up
5. monitor logs yeti-beat_1 errors without any interaction
6. issue GET to port opened via yeti to see errors for container yeti_1

Expected behavior

yeti-beat_1 does not crash
yeti_1 web app does not return 500 error

Actual behavior

yeti-beat_1 | Traceback (most recent call last):
yeti-beat_1 | File "/usr/local/bin/celery", line 10, in
yeti-beat_1 | sys.exit(main())
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/main.py", line 16, in main
yeti-beat_1 | _main()
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 322, in main
yeti-beat_1 | cmd.execute_from_commandline(argv)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 496, in execute_from_commandline
yeti-beat_1 | celery beat v4.3.0 (rhubarb) is starting.
yeti-beat_1 | super(CeleryCommand, self).execute_from_commandline(argv)))
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 298, in execute_from_commandline
yeti-beat_1 | return self.handle_argv(self.prog_name, argv[1:])
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 488, in handle_argv
yeti-beat_1 | return self.execute(command, argv)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 420, in execute
yeti-beat_1 | ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 302, in run_from_argv
yeti-beat_1 | sys.argv if argv is None else argv, command)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 386, in handle_argv
yeti-beat_1 | return self(*args, **options)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 252, in call
yeti-beat_1 | ret = self.run(*args, **kwargs)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/bin/beat.py", line 109, in run
yeti-beat_1 | return beat().run()
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 81, in run
yeti-beat_1 | self.start_scheduler()
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 100, in start_scheduler
yeti-beat_1 | print(self.banner(service))
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 122, in banner
yeti-beat_1 | c.reset(self.startup_info(service))),
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 132, in startup_info
yeti-beat_1 | scheduler = service.get_scheduler(lazy=True)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 627, in get_scheduler
yeti-beat_1 | lazy=lazy,
yeti-beat_1 | File "/opt/yeti/core/scheduling.py", line 61, in init
yeti-beat_1 | self.load_entries()
yeti-beat_1 | File "/opt/yeti/core/scheduling.py", line 79, in load_entries
yeti-beat_1 | self.loaded_entries = get_plugins()
yeti-beat_1 | File "/opt/yeti/core/yeti_plugins.py", line 31, in get_plugins
yeti-beat_1 | entry = obj.get_or_create(name=obj.default_values['name'])
yeti-beat_1 | File "/opt/yeti/core/database.py", line 90, in get_or_create
yeti-beat_1 | r = cls.objects(**select_dict).modify(upsert=True, **update_dict)
yeti-beat_1 | File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/base.py", line 650, in modify
yeti-beat_1 | raise OperationError(u'Update failed (%s)' % err)
yeti-beat_1 | mongoengine.errors.OperationError: Update failed (BSON field 'no_cursor_timeout' is an unknown field.)
docker_yeti-beat_1 exited with code 1

yeti_1 | ERROR:app:Exception on / [GET]
yeti_1 | Traceback (most recent call last):
yeti_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 2446, in wsgi_app
yeti_1 | response = self.full_dispatch_request()
yeti_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1944, in full_dispatch_request
yeti_1 | self.try_trigger_before_first_request_functions()
yeti_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1992, in try_trigger_before_first_request_functions
yeti_1 | func()
yeti_1 | File "./core/yeti_plugins.py", line 31, in get_plugins
yeti_1 | entry = obj.get_or_create(name=obj.default_values['name'])
yeti_1 | File "./core/database.py", line 90, in get_or_create
yeti_1 | r = cls.objects(**select_dict).modify(upsert=True, **update_dict)
yeti_1 | File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/base.py", line 650, in modify
yeti_1 | raise OperationError(u'Update failed (%s)' % err)
yeti_1 | OperationError: Update failed (BSON field 'no_cursor_timeout' is an unknown field.)
yeti_1 | [pid: 30|app: 0|req: 1/1] 165.225.32.118 () {52 vars in 937 bytes} [Fri Aug 23 18:54:30 2019] GET / => generated 290 bytes in 505 msecs (HTTP/1.1 500) 2 headers in 84 bytes (1 switches on core 0)
172.19.0.7:36972 (18 connections now open)
yeti_1 | ERROR:app:Exception on /favicon.ico [GET]
yeti_1 | Traceback (most recent call last):
yeti_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 2446, in wsgi_app
yeti_1 | response = self.full_dispatch_request()
yeti_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1944, in full_dispatch_request
yeti_1 | self.try_trigger_before_first_request_functions()
yeti_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1992, in try_trigger_before_first_request_functions
yeti_1 | func()
yeti_1 | File "./core/yeti_plugins.py", line 31, in get_plugins
yeti_1 | entry = obj.get_or_create(name=obj.default_values['name'])
yeti_1 | File "./core/database.py", line 90, in get_or_create
yeti_1 | r = cls.objects(**select_dict).modify(upsert=True, **update_dict)
yeti_1 | File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/base.py", line 650, in modify
yeti_1 | raise OperationError(u'Update failed (%s)' % err)
yeti_1 | OperationError: Update failed (BSON field 'no_cursor_timeout' is an unknown field.)

Thank you for your help!

[Mokatsu]

I manage to fix this error by following these steps:

cd <yeti git repo>/extras/docker
docker-compose down
docker system prune
docker volume prune

# edit the docker-compose.yaml or docker-compose-full.yaml
#change mango version from
# image:mongo:latest -> image:mongo:4.0

docker-compose up or docker-compose -f docker-compose-full.yaml up

Hope that Helps!

[starskizzle]

Mokatsu,

This does get the app started. Thank you. It may or may not be a long term fix. I am unable to start an investigation using an existing observable. Part of the stack trace includes mongodb. Please confirm if you experience the same behavior

yeti_1 | ERROR:app:Exception on /investigation/graph/observable/5d66cc1bce9120af951e68bf [GET] yeti_1 | Traceback (most recent call last): yeti_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 2446, in wsgi_app yeti_1 | response = self.full_dispatch_request() yeti_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1951, in full_dispatch_request yeti_1 | rv = self.handle_user_exception(e) yeti_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1820, in handle_user_exception yeti_1 | reraise(exc_type, exc_value, tb) yeti_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1949, in full_dispatch_request yeti_1 | rv = self.dispatch_request() yeti_1 | File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1935, in dispatch_request yeti_1 | return self.view_functions[rule.endpoint](**req.view_args) yeti_1 | File "/usr/local/lib/python2.7/dist-packages/flask_classy.py", line 200, in proxy yeti_1 | response = view(**request.view_args) yeti_1 | File "./core/web/helpers.py", line 39, in inner yeti_1 | return f(*args, **kwargs) yeti_1 | File "./core/web/frontend/investigations.py", line 47, in graph_node yeti_1 | investigation.add([], [node]) yeti_1 | File "./core/investigation.py", line 124, in add yeti_1 | self._node_changes('add', self.add_to_set, links, nodes) yeti_1 | File "./core/investigation.py", line 121, in _node_changes yeti_1 | self.modify(push__events=event, updated=datetime.utcnow()) yeti_1 | File "/usr/local/lib/python2.7/dist-packages/mongoengine/document.py", line 295, in modify yeti_1 | updated = self._qs(**query).modify(new=True, **update) yeti_1 | File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/base.py", line 657, in modify yeti_1 | result = self._document._from_son(result, only_fields=self.only_fields) yeti_1 | File "/usr/local/lib/python2.7/dist-packages/mongoengine/base/document.py", line 722, in _from_son yeti_1 | else field.to_python(value)) yeti_1 | File "/usr/local/lib/python2.7/dist-packages/mongoengine/base/fields.py", line 350, in to_python yeti_1 | for key, item in value.items()} yeti_1 | File "/usr/local/lib/python2.7/dist-packages/mongoengine/base/fields.py", line 350, in <dictcomp> yeti_1 | for key, item in value.items()} yeti_1 | File "/usr/local/lib/python2.7/dist-packages/mongoengine/fields.py", line 687, in to_python yeti_1 | return self.document_type._from_son(value, _auto_dereference=self._auto_dereference) yeti_1 | File "/usr/local/lib/python2.7/dist-packages/mongoengine/base/document.py", line 733, in _from_son yeti_1 | raise InvalidDocumentError(msg) yeti_1 | InvalidDocumentError: Invalid data to create a InvestigationEventinstance. yeti_1 | nodes - type object 'Node' has no attribute 'id' yeti_1 | [pid: 30|app: 0|req: 3/24] 165.225.32.79 () {54 vars in 1272 bytes} [Wed Aug 28 18:46:58 2019] GET /investigation/graph/observable/5d66cc1bce9120af951e68bf => generated 290 bytes in 303 msecs (HTTP/1.1 500) 2 headers in 84 bytes (1 switches on core 0)

[zetahoq]

I had similar issues with Flask and mongo. I was already planning to use a Caddy for the web server so i eliminated command: ['docker-entrypoint.sh', 'uwsgi-http'] from the Yeti service and changed the port back to 5000

With Mongo I changed the version and defined a persistence volume.

Below are excerpts from my yml file

 yeti:
    build:
      context: ../../
      dockerfile: ./extras/docker/Dockerfile
    image: yeti-platform
    ports:
      - "5000:5000"
    links:
      - redis
      - mongodb
    volumes:
      - /data/yeti.conf:/opt/yeti/yeti.conf:ro
    restart: always
    depends_on:
      - redis
      - mongodb

  mongodb:
    image: mongo:4.0.12
    volumes:
      - /data/db:/data/db
    environment:
      - MONGO_LOG_DIR=/dev/null
    command: mongod

[tomchop]

Thanks for debugging this folks! I'll change the mongodb image to use 4.0 and not latest. (other people are also experiencing this bug: MongoEngine/mongoengine#2148)