Rework agents installation

[gazarenkov]

The problems:

• Current mechanism of Agents running requires too many interactions with master and for some cases per-agent specific/custom "launching" java code.
• As a consequence of unpredictable big amount of interactions with master it may generate big log traffic which may significantly impact on master and client loading/workability (up to make it broken). The problem like we used to have with command execs.

Solution:
Make small (as small as possible, Golang app) agent-installer which will be injected first all the time when machine requires Agents. I.e. any time when workspace -> environment -> machine section contains one or more agent entry. Responsibilities of agent-installer:

• to run agent script locally on machine
• to orchestrate (they may not be installed simultaneously to avoid conflicts, only one-by-one) multiple agents's launching using applicable criteria to check if agent installed/launched such as:
  -- check if bash scripts process completed (installation only usecase) if no server launching supposed (agent knows that)
  -- check if server(s) launched (installation and launching usecase)
• to send machine log messages via websocket to interested sides
• to send notifications about starting/completing/failing during agent installation to interested sides.

@garagatyi @tolusha let's make a plan to replace old mechanism as discussed.

• Agents installation specification Agents installation specification #4096
• Refactor exec + terminal agent to separate components Refactor exec + terminal agent to separate components #4097
• Move exec-agent/process module to core/process Move exec-agent/process module to core/process #5131
• Add process exit code to the core/process process_died event Add process exit code to the core/process process_died event #5147
• Remove core/process -> core/rpc dependency Remove core/process -> core/rpc dependency  #5149
• Separate exec and terminal agent Separate exec and terminal agents #4098
• Define agents configuration file format Define agents configuration file format #5151
• Add ability to connect to jsonrpc websocket endpoint using core/rpc Add ability to connect to jsonrpc websocket endpoint using core/rpc #5150
• Review and define agent bootstrapper events Review and define agent bootstrapper events #5266
• Implement bootstrappert Implement bootstrapper #4099
• Rework WS start back end code to use new install agent API for workspace start Rework docker runtime infrastructure to use bootstrapper events during workspace start #4101
• Get rid of java agent launchers Get rid of java agent launchers #5413

[garagatyi]

I suggest next todo list which can be source of tasks for this epic:

1. Create specification

• install agent workflow
• install agent API (how client should connect to it, stream and interpret data)

2. Refactor exec + terminal agent to separate components needed for:

• commons(place for utilities and structures like FileLogger)
• commands execution(execute cmd + subscriber interface for logs consumption)
• authentication
• transport
  • REST(routing + request handling)
  • WS + JSONRPC engine(routing + request handling + channel management)

etc. It should be possible to create new agent that can reuse these components.
3. Separate exec and terminal agents.
4. Create install agent that reuses components created in previous step.
5. Create java client for install-agent API.
6. Rework WS start back end code to use new install agent API for workspace start.
7. Rework GWT, UD clients to use new install agent API for logs streaming/downloading.

[dyrone]

@garagatyi 👍

how is the process of the authentication?

[skabashnyuk]

Hi @dyrone
I think most parts of authentification would be covered by this epic #5362