Baf 849/fleet management

[MarioIvancik]

Task:
https://youtrack.bringauto.com/issue/BAF-849/Add-Fleet-Management-Integration-Lay.-to-etna

[Nothrax]

scripts/requirements.txt is missing dependencies pydantic and python-dateutil, after installing missing deps I get an error running the fleet init:

Traceback (most recent call last):
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/connection.py", line 238, in connect
    self.sock = self._new_conn()
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/connection.py", line 213, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fa0ca68cac0>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/nothrax/Documents/bringauto/etna/scripts/fleet-init/main.py", line 100, in <module>
    main()
  File "/home/nothrax/Documents/bringauto/etna/scripts/fleet-init/main.py", line 86, in main
    delete_all(api_client)
  File "/home/nothrax/Documents/bringauto/etna/scripts/fleet-init/fleet/query/utils.py", line 12, in delete_all
    orders = order_api.get_orders()
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/pydantic/validate_call_decorator.py", line 59, in wrapper_function
    return validate_call_wrapper(*args, **kwargs)
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/pydantic/_internal/_validate_call.py", line 81, in __call__
    res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/fleet_management_http_client_python/api/order_api.py", line 1233, in get_orders
    response_data = self.api_client.call_api(
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/fleet_management_http_client_python/api_client.py", line 270, in call_api
    response_data = self.rest_client.request(
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/fleet_management_http_client_python/rest.py", line 244, in request
    r = self.pool_manager.request(
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/_request_methods.py", line 136, in request
    return self.request_encode_url(
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/_request_methods.py", line 183, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/poolmanager.py", line 444, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    return self.urlopen(
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    return self.urlopen(
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    return self.urlopen(
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
  File "/home/nothrax/Documents/bringauto/etna/scripts/venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8081): Max retries exceeded with url: /v2/management/order?api_key=ManagementStaticAccessKey (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa0ca68cac0>: Failed to establish a new connection: [Errno 111] Connection refused'))

Fleet HTTP api was not running, better log would be appreciated, or common issues section in readme

[Nothrax]

while running etna with profile "all" I get an error 024-05-18 07:27:52.358 UTC [42] FATAL: database "management_api" does not exist and 2024-05-18 07:27:53.298 UTC [43] FATAL: database "protocol_api" does not exist

I had to delete existing posgress container - common issues section in readme or force creation of the databases

[Nothrax]

When I was fixed all issues with file permissions and was able to start profile "all" the module gateway did not connect to external server, so no data were available using http api, there was an error

external-server-1      | [08:52:44] ERROR    ExternalServer: Unexpected error      external_server.py:105
external-server-1      |                     occurred: 'NoneType' object has no                          
external-server-1      |                     attribute 'HasField'                                        
external-server-1      | [08:52:46] INFO     ExternalServer: Starting the connect  external_server.py:111
external-server-1      |                     sequence                                                    
external-server-1      |            INFO     MqttClient: Server connected to MQTT       mqtt_client.py:89
external-server-1      |                     broker                                                      
external-server-1      |            INFO     ExternalServer: Expecting a connect   external_server.py:119
external-server-1      |                     message       

[Nothrax]

Expectation: I have machine with docker compose, I download this repo, update submodules, run profile "all" and I can see data in http and management api, before that, this cannot be merged.