Deployed Server doesn't have a worker for the services queue

[gaspardc-met]

Hello clearml team,
First: thank you for the great work, and for open-sourcing most of it !

I have a question about the workers/queues on a self-hosted server.

My setup:

• I deployed the Server to a self-hosted Kubernetes cluster via the helm charts (it was quick and easy, thanks !)
  https://github.com/allegroai/clearml-helm-charts#usage=
• I cloned the clearml repo to run the examples at https://github.com/allegroai/clearml/tree/master/examples/pipeline
• Of course I logged into the server with clearml-init, without issues
• What I wanted to test was: execute a full pipeline remotely (controller logic and steps)
• I used pipeline_from_decorators and commented this line to run controller and steps remotely: https://github.com/allegroai/clearml/blob/6af94ccf7b5ad39a8ea09072b197ab889b9a9c21/examples/pipeline/pipeline_from_decorator.py#L96
• Note that the same problem pops up with pipeline_from_functions.py
• I run the module locally

Now the pipeline is pending on the Server UI, and when I go to Workers & Queues I see that I have an idle worker k8s-agent set to queue default and no worker for queue services

My point being, if the services queue is important, why doesn't the Server/Agent set a worker for it by default ?
Why isn't possible to move the job from the services to the default queue successfully in the UI ?
Can I force the pipeline controller to run with the default queue along with the steps ? I thought it would be possible using pipeline_execution_queue="default" in the decorator, but doesn't seem to work on my side.

Thanks for your feedback

[jkhenning]

@gaspard-met you're quite right - we're planning to add this to one of the next versions, it basically requires some changes to the default docker-compose and some startup automation to support it 🙂

[hotshotdragon]

Hi @jkhenning , I am facing the same issue, but I am not self hosting the server. Console is stuck at
Launching step: step_one Parameters: {'kwargs/pickle_url': 'https://github.com/***/data.pkl'} Configurations: {} Overrides: {}

when checking the queue, it is showing as pending.

[jkhenning]

Hi @hotshotdragon,

If you are not hosting the server, which server is it?