gitlab runner fails at with Error: GitLab API not found

[GeeCastro]

I am trying to test CML on a locally deployed GitLab server. I setup the token which seems to work. However, the pipeline fails due to an API error on "cml-send-comment"

Running with gitlab-runner 13.10.0 (54944146)
  on dev01 XXXXXXX
section_start:1618225417:prepare_executor
Preparing the "docker" executor
Using Docker executor with image dvcorg/cml-py3:latest ...
Using locally found image version due to "if-not-present" pull policy
Using docker image sha256:1ace1dc1b2a0543e4a7a9b34733456f5b69acb9b658d9165f912e7b2c0661dfc for dvcorg/cml-py3:latest with digest dvcorg/cml-py3@sha256:54bc3c205f7d643b83807915a752f2447c0d85790035e5d524290bfbc8966373 ...
section_end:1618225420:prepare_executor
section_start:1618225420:prepare_script
Preparing environment
Running on runner-xfdxembn-project-2-concurrent-0 via a250b8db9876...
section_end:1618225422:prepare_script
section_start:1618225422:get_sources
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/root/cml-test/.git/
Checking out c121223b as master...
Removing report.md

Skipping Git submodules setup
section_end:1618225424:get_sources
section_start:1618225424:step_script
Executing "step_script" stage of the job script
Using docker image sha256:1ace1dc1b2a0543e4a7a9b34733456f5b69acb9b658d9165f912e7b2c0661dfc for dvcorg/cml-py3:latest with digest dvcorg/cml-py3@sha256:54bc3c205f7d643b83807915a752f2447c0d85790035e5d524290bfbc8966373 ...
$ echo 'Hi from CML!' >> report.md
$ cml-send-comment report.md
Error: GitLab API not found
    at Gitlab.repo_base (/cml/src/drivers/gitlab.js:55:37)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Gitlab.project_path (/cml/src/drivers/gitlab.js:25:23)
    at async Gitlab.comment_create (/cml/src/drivers/gitlab.js:63:26)
    at async CML.comment_create (/cml/src/cml.js:83:12)
    at async run (/cml/bin/cml-send-comment.js:20:3)
section_end:1618225426:step_script
section_start:1618225426:cleanup_file_variables
Cleaning up file based variables
section_end:1618225427:cleanup_file_variables
ERROR: Job failed: exit code 1

my .gitlab-ci.yml files looks like this

stages: 
    - cml_run
    
cml:
    stage: cml_run
    image: dvcorg/cml-py3:latest
    
    script:
      - echo 'Hi from CML!' >> report.md
      - cml-send-comment report.md

[0x2b3bfa0]

👋 Welcome, @Chichilele! Can you please share your non-sensitive environment variables? It could not be detecting your repository URL correctly.

Alternatively, please try running the command with the --repo https://gitlab.com/user/project where gitlab.com is your GitLab server, user is your user name and project is the repository where you're trying to publish a comment.

[GeeCastro]

Please find my environment below (output of set within the runner). I am running this as the root/admin user in GitLab. It's freshly deployed on docker swarm on a private network, with self-signed certificates. The runner reaches the server and clones the repo so I assume everything works fine so far regarding networking and certs.

Click to expand!

BASH=/bin/bash
BASHOPTS=cmdhist:complete_fullquote:extquote:force_fignore:hostcomplete:interactive_comments:progcomp:promptvars:sourcepath
BASH_ALIASES=()
BASH_ARGC=()
BASH_ARGV=()
BASH_CMDS=()
BASH_LINENO=()
BASH_SOURCE=()
BASH_VERSINFO=([0]="4" [1]="4" [2]="20" [3]="1" [4]="release" [5]="x86_64-pc-linux-gnu")
BASH_VERSION='4.4.20(1)-release'
CI=true
CI_API_V4_URL=https://gitlab/api/v4
CI_BUILDS_DIR=/builds
CI_BUILD_BEFORE_SHA=XXXX01
CI_BUILD_ID=15
CI_BUILD_NAME=cml
CI_BUILD_REF=XXXX02
CI_BUILD_REF_NAME=master
CI_BUILD_REF_SLUG=master
CI_BUILD_STAGE=cml_run
CI_BUILD_TOKEN=[MASKED]
CI_COMMIT_BEFORE_SHA=XXXX01
CI_COMMIT_BRANCH=master
CI_COMMIT_DESCRIPTION=
CI_COMMIT_MESSAGE='Update .gitlab-ci.yml'
CI_COMMIT_REF_NAME=master
CI_COMMIT_REF_PROTECTED=true
CI_COMMIT_REF_SLUG=master
CI_COMMIT_SHA=XXXX02
CI_COMMIT_SHORT_SHA=XXXX03
CI_COMMIT_TIMESTAMP=2021-04-12T12:12:53+00:00
CI_COMMIT_TITLE='Update .gitlab-ci.yml'
CI_CONCURRENT_ID=0
CI_CONCURRENT_PROJECT_ID=0
CI_CONFIG_PATH=.gitlab-ci.yml
CI_DEFAULT_BRANCH=master
CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX=gitlab:443/root/dependency_proxy/containers
CI_DEPENDENCY_PROXY_PASSWORD=[MASKED]
CI_DEPENDENCY_PROXY_SERVER=gitlab:443
CI_DEPENDENCY_PROXY_USER=gitlab-ci-token
CI_DISPOSABLE_ENVIRONMENT=true
CI_JOB_ID=15
CI_JOB_IMAGE=dvcorg/cml-py3:latest
CI_JOB_JWT=[MASKED]
CI_JOB_NAME=cml
CI_JOB_STAGE=cml_run
CI_JOB_STARTED_AT=2021-04-12T12:12:54Z
CI_JOB_STATUS=running
CI_JOB_TOKEN=[MASKED]
CI_JOB_URL=https://gitlab/root/cml-test/-/jobs/15
CI_NODE_TOTAL=1
CI_PAGES_DOMAIN=example.com
CI_PAGES_URL=http://root.example.com/cml-test
CI_PIPELINE_CREATED_AT=2021-04-12T12:12:53Z
CI_PIPELINE_ID=4
CI_PIPELINE_IID=4
CI_PIPELINE_SOURCE=push
CI_PIPELINE_URL=https://gitlab/root/cml-test/-/pipelines/4
CI_PROJECT_CONFIG_PATH=.gitlab-ci.yml
CI_PROJECT_DIR=/builds/root/cml-test
CI_PROJECT_ID=2
CI_PROJECT_NAME=cml-test
CI_PROJECT_NAMESPACE=root
CI_PROJECT_PATH=root/cml-test
CI_PROJECT_PATH_SLUG=root-cml-test
CI_PROJECT_REPOSITORY_LANGUAGES=
CI_PROJECT_ROOT_NAMESPACE=root
CI_PROJECT_TITLE=CML-test
CI_PROJECT_URL=https://gitlab/root/cml-test
CI_PROJECT_VISIBILITY=private
CI_REGISTRY=gitlab:4567
CI_REGISTRY_IMAGE=gitlab:4567/root/cml-test
CI_REGISTRY_PASSWORD=[MASKED]
CI_REGISTRY_USER=gitlab-ci-token
CI_REPOSITORY_URL=https://gitlab-ci-token:[MASKED]@gitlab/root/cml-test.git
CI_RUNNER_DESCRIPTION=dev01
CI_RUNNER_EXECUTABLE_ARCH=linux/amd64
CI_RUNNER_ID=1
CI_RUNNER_REVISION=54944146
CI_RUNNER_SHORT_TOKEN=xFDxeMBN
CI_RUNNER_TAGS='cml, gpu'
CI_RUNNER_VERSION=13.10.0
CI_SERVER=yes
CI_SERVER_HOST=gitlab
CI_SERVER_NAME=GitLab
CI_SERVER_PORT=443
CI_SERVER_PROTOCOL=https
CI_SERVER_REVISION=cc4224220e6
CI_SERVER_TLS_CA_FILE=/builds/root/cml-test.tmp/CI_SERVER_TLS_CA_FILE
CI_SERVER_URL=https://gitlab
CI_SERVER_VERSION=13.10.2-ee
CI_SERVER_VERSION_MAJOR=13
CI_SERVER_VERSION_MINOR=10
CI_SERVER_VERSION_PATCH=2
CUDA_PKG_VERSION=10-1=10.1.243-1
CUDA_VERSION=10.1.243
DEBIAN_FRONTEND=noninteractive
DIRSTACK=()
EUID=0
FF_CMD_DISABLE_DELAYED_ERROR_LEVEL_EXPANSION=false
FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR=false
FF_ENABLE_BASH_EXIT_CODE_CHECK=false
FF_GITLAB_REGISTRY_HELPER_IMAGE=false
FF_NETWORK_PER_BUILD=false
FF_RESET_HELPER_IMAGE_ENTRYPOINT=true
FF_SHELL_EXECUTOR_USE_LEGACY_PROCESS_KILL=false
FF_SKIP_NOOP_BUILD_STAGES=true
FF_USE_DIRECT_DOWNLOAD=true
FF_USE_FASTZIP=false
FF_USE_GO_CLOUD_WITH_CACHE_ARCHIVER=true
FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY=true
FF_USE_WINDOWS_LEGACY_PROCESS_STRATEGY=true
GITLAB_CI=true
GITLAB_FEATURES=
GITLAB_USER_EMAIL=admin@example.com
GITLAB_USER_ID=1
GITLAB_USER_LOGIN=root
GITLAB_USER_NAME=Administrator
GO111MODULE=auto
GROUPS=()
HOME=/root
HOSTNAME=runner-xfdxembn-project-2-concurrent-0
HOSTTYPE=x86_64
IFS=$' \t\n'
IN_DOCKER=1
LANG=en_US.UTF-8
LANGUAGE=en_US:en
LC_ALL=en_US.UTF-8
LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
MACHTYPE=x86_64-pc-linux-gnu
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_REQUIRE_CUDA='cuda>=10.1 brand=tesla,driver>=396,driver<397 brand=tesla,driver>=410,driver<411 brand=tesla,driver>=418,driver<419'
NVIDIA_VISIBLE_DEVICES=all
OLDPWD=/home/runner
OPTERR=1
OPTIND=1
OSTYPE=linux-gnu
PATH=/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PIPESTATUS=([0]="0")
PPID=0
PS4='+ '
PWD=/builds/root/cml-test
RUNNER_ALLOW_RUNASROOT=1
RUNNER_PATH=/home/runner
SHELL=/bin/bash
SHELLOPTS=braceexpand:errexit:hashall:interactive-comments:pipefail
SHLVL=1
TERM=dumb
UID=0
_=$'\E[32;1m$ set >> report.md\E[0;m'
repo_token=[MASKED]

[DavidGOrtega]

👋 @Chichilele

Can you try?

stages: 
    - cml_run
    
cml:
    stage: cml_run
    image: dvcorg/cml-py3:latest
    
    script:
      - echo 'Hi from CML!' >> report.md
      - cml-send-comment report.md --repo=https://gitlab/root/cml-test

[GeeCastro]

Yep, I did try with the repo and it didn't help unfortunately

[GeeCastro]

I was directly updating the master branch and thought writing a comment with no merge request may fail. So I created a new branch + merge request but I still get the same error. Is there a way to make the error more verbose? I am not familiar with js

[DavidGOrtega]

Could you please add this in the workflow and tell me the version?

stages: 
    - cml_run
    
cml:
    stage: cml_run
    image: dvcorg/cml-py3:latest
    
    script:
      - cml-send-comment --version
      
      - echo 'Hi from CML!' >> report.md
      - cml-send-comment report.md --repo=https://gitlab/root/cml-test

[GeeCastro]

cml-send-comment --version
0.3.5

There is a recent merged PR about gitlab. And I believe the image I am using is of the same date

[DavidGOrtega]

According to your logs you endpoint should be gitlab, can you please try these two calls in any shell with network access to your GL instance?

curl --header "PRIVATE-TOKEN: YOUR_GITLAB_TOKEN" "https://gitlab/api/v4/version"

curl --header "PRIVATE-TOKEN: YOUR_GITLAB_TOKEN" "https://gitlab/root/api/v4/version"

[DavidGOrtega]

Or in the. workflow

stages: 
    - cml_run
    
cml:
    stage: cml_run
    image: dvcorg/cml-py3:latest
    
    script:
      - curl --header "PRIVATE-TOKEN: $repo_token" "https://gitlab/api/v4/version"
      - curl --header "PRIVATE-TOKEN: $repo_token" "https://gitlab/root/api/v4/version"
     
      - echo 'Hi from CML!' >> report.md
      - cml-send-comment report.md --repo=https://gitlab/root/cml-test

[GeeCastro]

➜  ~ docker exec s2_gitlab.1.XXX curl --insecure --header "PRIVATE-TOKEN: XXXXXX" "https://gitlab/root/api/v4/version"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    94  100    94    0     0   1620      0 --:--:-- --:--:-- --:--:--  1620
<html><body>You are being <a href="https://gitlab/users/sign_in">redirected</a>.</body></html>%
➜  ~ docker exec s2_gitlab.1.XXX curl --insecure --header "PRIVATE-TOKEN: XXXXXX" "https://gitlab/api/v4/version"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    30  100    30    0     0    714      0 --:--:-- --:--:-- --:--:--   714
{"message":"401 Unauthorized"}%

[DavidGOrtega]

@Chichilele Your token seems to not be valid, could you please verify it ?

@0x2b3bfa0 we have to check this case also since we are checking this later on having a proper api endpoint

[GeeCastro]

From within GitLab CI

$ curl --insecure --header "PRIVATE-TOKEN: $repo_token" "https://gitlab/api/v4/version" > api.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    49  100    49    0     0    636      0 --:--:-- --:--:-- --:--:--   636
$ cat api.txt
{"version":"13.10.2-ee","revision":"cc4224220e6"}
$ curl --insecure --header "PRIVATE-TOKEN: $repo_token" "https://gitlab/root/api/v4/version" > apiuser.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    94  100    94    0     0   1402      0 --:--:-- --:--:-- --:--:--  1402
$ cat apiuser.txt
<html><body>You are being <a href="https://gitlab/users/sign_in">redirected</a>.</body></html>

[DavidGOrtega]

@Chichilele also probably you should setup this enviroonment variable to overcome that insecure SSL in the workflow or in the CI as env variable

export NODE_TLS_REJECT_UNAUTHORIZED=0