Setup issues with release 0.40.0 in K8s and installation with Helm

[odedsCoolScripts]

HI Guys,

Thanks for the new features in version 0.40.0

Although, I can say that I'm having a very hard time to set CDS up no my k8s cluster.
I'll describe my setup steps and point where thing gone wrong:

1. I have a k8s cluster on aws-eks

2. Helm, kubectl, tiller and aws cli installed and configured correctly on my Mac

3. cloned the git repo - https://github.com/ovh/cds.git to my home directory

4. inside ~/cds, git checkout 0.40.0. Now I'm in release 0.40. 0

5. in contrib/helm/cds, installed CDS with helm
   helm dependency update
   helm install --name cds-weka .
   Issue 1:
   in helm/cds directory, tag values in values.yaml for cds image are 0.39.2, therefore, the installation resulted with older version - 0.39.2. 'app version' in Charts.yaml is also 0.39.2

6. I deleted the helm installation- 'helm delete cds-weka --purge'

7. modified values.yaml and Charts.yaml to the correct version -0.40.0

8. helm dependency update
   helm install --name cds-weka .
   Now, version 0.40.0 is installed.

9. added the bitbucketcloud section to config.toml, cluentid and secret and delete api and vcs pods. exactly as written in this guide:
   https://ovh.github.io/cds/docs/integrations/bitbucketcloud/
   issue 2:
   open the cds ui--> enter a project advanced tab--> see no 'bitbucketcloud' in the 'Link to a repository manager' drop-down-list.
   I think there's missing documentation for how to enable the service.

10. Go back to condig.toml- give a name to the 'name' field in 'CDS VCS Settings'- cds-weka-cds-vcs.

11. kill api and vcs pods

12. Finally, see bitbucketcloud in project's advanced tab.

13. Connect to bitbucket--> 'grant access' in the bitbucket redirected page.

14. Enter the code that is found url to 'validate' field in cds UI

15. see the message - linked successfully to repository.

16. now, trying to synchronise my repo list (drop down list) leads to:
    issue 3:
    the repositories drop down list is empty! authentication to my bitbucket cloud account probably not authorized...
    looking at the api pod logs, I see this:
    Reading configuration file config.toml 2019-07-02 07:32:36 [INFO] Directory /app/keys has been created 2019-07-02 07:32:36 [INFO] api> Service registered 2019-07-02 07:32:36 [INFO] Starting CDS API Server 0.40.0+cds.9951 2019-07-02 07:32:36 [INFO] Initializing mail driver... 2019-07-02 07:32:36 [INFO] Initializing feature flipping with izanami 2019-07-02 07:32:36 [INFO] Initializing local objectstore... 2019-07-02 07:32:36 [INFO] ObjectStore> Initialize Filesystem driver on directory: /app/artifacts 2019-07-02 07:32:36 [INFO] Initializing database connection... Starting service api 2019-07-02 07:32:36 [INFO] Bootstrapping database... 2019-07-02 07:32:36 [INFO] Initializing redis cache on weka-cds-devops-redis-master:6379... 2019-07-02 07:32:36 [INFO] Initializing HTTP router 2019-07-02 07:32:36 [INFO] Initializing Events broker 2019-07-02 07:32:36 [INFO] api> Stats initialized 2019-07-02 07:32:36 [INFO] Initializing Metrics 2019-07-02 07:32:36 [INFO] api> Metrics initialized 2019-07-02 07:32:36 [INFO] Initializing Authentication driver... 2019-07-02 07:32:36 [INFO] Auth> Initializing driver (local) 2019-07-02 07:32:36 [INFO] Auth> Connecting to session store 2019-07-02 07:32:36 [INFO] Initializing event broker... 2019-07-02 07:32:36 [INFO] Initializing internal routines... 2019-07-02 07:32:36 [INFO] Migration> CleanArtifactBuiltinActions> Already done (status: DONE) 2019-07-02 07:32:36 [INFO] Migration> WorkflowOldStruct> Already done (status: DONE) 2019-07-02 07:32:36 [INFO] Starting GRPC services on :8082 2019-07-02 07:32:36 [INFO] Migration> StageConditions> Already done (status: DONE) 2019-07-02 07:32:36 [INFO] Migration> WorkflowNotification> Already done (status: DONE) 2019-07-02 07:32:36 [INFO] Migration> ActionModelRequirements> Already done (status: DONE) 2019-07-02 07:32:36 [INFO] Starting CDS API HTTP Server on :8081 2019-07-02 07:35:06 [WARN] pprofLabel>recoverWrap>Handle>attachRepositoriesManagerHandler: repository not found (caused by: attachRepositoriesManager> Cannot get repo undefined: pprofLabel>recoverWrap>Handle>attachRepositoriesManagerHandler>RepoByFullname>doJSONRequest: resource not found (caused by: unable to get repo undefined from bitbucketcloud: pprofLabel>recoverWrap>Handle>attachRepositoriesManagerHandler>RepoByFullname>doJSONRequest>DoJSONRequest>doJSONRequest>ErrorWithFallback>NewErrorWithStack: wrong request (from: Unable to perform request on service weka-cds-devops-cds-vcs (vcs)) (caused by: Request Failed))) error_uuid=e9769dc1-9c9b-11e9-bd2e-76f3b50cd5a9 method=POST request_uri=/project/DEVOPTE/repositories_manager/bitbucketcloud/application/devops-test-app/attach?fullname=undefined stack_trace=attachRepositoriesManager> Cannot get repo undefined: pprofLabel>recoverWrap>Handle>attachRepositoriesManagerHandler>RepoByFullname>doJSONRequest: resource not found (caused by: unable to get repo undefined from bitbucketcloud: pprofLabel>recoverWrap>Handle>attachRepositoriesManagerHandler>RepoByFullname>doJSONRequest>DoJSONRequest>doJSONRequest>ErrorWithFallback>NewErrorWithStack: wrong request (from: Unable to perform request on service weka-cds-devops-cds-vcs (vcs)) (caused by: Request Failed)) github.com/ovh/cds/sdk.WrapError /go/src/github.com/ovh/cds/sdk/error.go:732 github.com/ovh/cds/engine/api.(*API).attachRepositoriesManagerHandler.func1 /go/src/github.com/ovh/cds/engine/api/repositories_manager.go:476 github.com/ovh/cds/engine/api.(*Router).Handle.func1 /go/src/github.com/ovh/cds/engine/api/router.go:245 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:1995 github.com/ovh/cds/engine/api.(*Router).recoverWrap.func1 /go/src/github.com/ovh/cds/engine/api/router.go:152 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:1995 github.com/ovh/cds/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1 /go/src/github.com/ovh/cds/vendor/github.com/gorilla/handlers/compress.go:143 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:1995 github.com/ovh/cds/engine/api.(*Router).pprofLabel.func1 /go/src/github.com/ovh/cds/engine/api/router.go:89 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:1995 github.com/ovh/cds/vendor/github.com/gorilla/mux.(*Router).ServeHTTP /go/src/github.com/ovh/cds/vendor/github.com/gorilla/mux/mux.go:114 net/http.serverHandler.ServeHTTP

I tried many times since then, couldn't fix it.
Please assist guys, thank you for your effort.

[yesnault]

Hi @odedsCoolScripts, thank you for this detailed report.

Issue 1: #4431
issues 2 & 3, we investigate.

[odedsCoolScripts]

Thanks @yesnault
Any updated upon that?

Regards

[bnjjj]

For your issue 2, it's a lack of documentation, fixed here #4459

I will now make the same steps as you to reproduce the bug and investigate. Just to know, when you say you updated the config.toml. Do you do that with the config.toml on your local desk and then push via helm upgrade ? Or directly edit in the pod ?

[odedsCoolScripts]

I edited through my desk. First option

Thanks

[bnjjj]

Hello @odedsCoolScripts I come back to you. I try to reproduce your issue but without "success". What I can recommend is to try to delete and re install a brand new CDS with helm install with the bitbucketcloud configuration already in your config.toml.

For your information in my config.toml I just put this:

[vcs.servers.bitbucketcloud]

      [vcs.servers.bitbucketcloud.bitbucketcloud]

        # Bitbucket Cloud OAuth Key
        clientId = "XXX"

        # Bitbucket Cloud OAuth Secret
        clientSecret = "XXX"

        # Does webhooks are supported by VCS Server
        disableWebHooks = false

        # Does webhooks creation are supported by VCS Server
        disableWebHooksCreation = false

        #proxyWebhook = "https://myproxy.com/"

        [vcs.servers.bitbucketcloud.bitbucketcloud.Status]

          # Set to true if you don't want CDS to push statuses on the VCS server
          disable = false

          # Set to true if you don't want CDS to push CDS URL in statuses on the VCS server
          showDetail = false

at this line https://github.com/ovh/cds/blob/master/contrib/helm/cds/config.toml#L506 with good clientId and clientSecret

[odedsCoolScripts]

Thanks a lot.

Working in that now

[odedsCoolScripts]

Thanks a lot for the support.

I tried many times, linked to bitbucket successfully (as far as I can see in the GUI green message) but I see no repositories in the drop down list:

I'm uploading my config file, replaced the credentials with string

config.toml.debug.txt

BTW, After pressing on 'grant access', I'm redirected to a blank page with on line:
{"id":11,"message":"forbidden","data":null,"uuid":"98c01faf-a96d-11e9-ad80-baef22e44f68"}
is that OK? the page has the code in the URL, example:
http://a8217dd47a96c11e99a3502265ed1aeb-368836905.eu-west-1.elb.amazonaws.com/cdsapi/repositories_manager/oauth2/callback?**code=CXUkxHZKpF5rXXXXXX**
so I used it to paste in 'validate' field. Is that the right flow? what is the meaning 'forbidden' here?

error logs from api pod:
2019-07-18 15:06:19 [WARN] pprofLabel>recoverWrap>Handle>repositoriesManagerOAuthCallbackHandler: forbidden (caused by: repositoriesManagerAuthorizeCallback> Error) error_uuid=98c01faf-a96d-11e9-ad80-baef22e44f68 method=GET request_uri=/repositories_manager/oauth2/callback?code=CXUkxHZKpF5rGvR69n stack_trace=repositoriesManagerAuthorizeCallback> Error github.com/ovh/cds/sdk.WrapError /go/src/github.com/ovh/cds/sdk/error.go:732 github.com/ovh/cds/engine/api.(*API).repositoriesManagerOAuthCallbackHandler.func1 /go/src/github.com/ovh/cds/engine/api/repositories_manager.go:115 github.com/ovh/cds/engine/api.(*Router).Handle.func1 /go/src/github.com/ovh/cds/engine/api/router.go:245 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:1995 github.com/ovh/cds/engine/api.(*Router).recoverWrap.func1 /go/src/github.com/ovh/cds/engine/api/router.go:152 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:1995 github.com/ovh/cds/vendor/github.com/gorilla/handlers.CompressHandlerLevel.func1 /go/src/github.com/ovh/cds/vendor/github.com/gorilla/handlers/compress.go:143 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:1995 github.com/ovh/cds/engine/api.(*Router).pprofLabel.func1 /go/src/github.com/ovh/cds/engine/api/router.go:89 net/http.HandlerFunc.ServeHTTP /usr/local/go/src/net/http/server.go:1995 github.com/ovh/cds/vendor/github.com/gorilla/mux.(*Router).ServeHTTP /go/src/github.com/ovh/cds/vendor/github.com/gorilla/mux/mux.go:114 net/http.serverHandler.ServeHTTP /usr/local/go/src/net/http/server.go:2774 net/http.(*conn).serve /usr/local/go/src/net/http/server.go:1878 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1337 status=403 2019-07-18 15:06:35 [ERROR] PushInElasticSearch> Unable to send event sdk.EventProjectVCSServerAdd to elasticsearch [500]: GoRoutine>Serve>PushInElasticSearch>DoJSONRequest>doJSONRequest>ErrorWithFallback>NewErrorWithStack: wrong request (from: Unable to perform request on service cds-weka-cds-elasticsearch (elasticsearch)) (caused by: internal server error) 2019-07-18 15:06:35 [ERROR] PushInElasticSearch> Unable to send event sdk.EventWarningAdd to elasticsearch [500]: GoRoutine>Serve>PushInElasticSearch>DoJSONRequest>doJSONRequest>ErrorWithFallback>NewErrorWithStack: wrong request (from: Unable to perform request on service cds-weka-cds-elasticsearch (elasticsearch)) (caused by: internal server error) 2019-07-18 15:06:54 [ERROR] PushInElasticSearch> Unable to send event sdk.EventApplicationAdd to elasticsearch [500]: GoRoutine>Serve>PushInElasticSearch>DoJSONRequest>doJSONRequest>ErrorWithFallback>NewErrorWithStack: wrong request (from: Unable to perform request on service cds-weka-cds-elasticsearch (elasticsearch)) (caused by: internal server error)
Thanks again,

Oded

[bnjjj]

Can you send me also the logs from vcs service and elasticsearch service please ? Via https://pastebin.com/ in order to have a better view of your logs :)

[odedsCoolScripts]

Thanks a lot @bnjjj
Logs are here (I put a title head before each log):
https://pastebin.com/MPgJyVRa

[odedsCoolScripts]

Hi everyone, any update?

[bnjjj]

Ok I spotted the bug which gives you a 403 Forbidden and force you to indicate the code. But I don't know if it will fix your bug about empty list of repositories because I can't reproduce with a new fresh install on a k8s cluster. BTW it seems your elasticsearch pods are not correctly spawned. Here is the corresponding PR: #4496

[mckunda]

I have the same issue 3 as OP. I'm running on Windows 10, downloaded binary cds-engine-windows-amd64.exe from here (0.40.0). Toml config is local. Running only api, vcs and hooks services.

[bnjjj]

a new release will be available this week to fix this issue

[yesnault]

https://github.com/ovh/cds/releases/tag/0.41.0