New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

404 not found on /api/v1/charts calls #422

Closed
cabrinoob opened this Issue Feb 12, 2018 · 12 comments

Comments

Projects
None yet
5 participants
@cabrinoob

cabrinoob commented Feb 12, 2018

Hi,
I have a strange problem with the monocular API. Everything seems to work fine in k8s, but when I open the UI I have a spinner with an error in console :

Failed to load resource: the server responded with a status of 404 () on /api/v1/charts

I checked everything in my K8S dashboard, no error logs. Ingress rules seems ok ....

The only modification I've done was to remove the readinessProbe test on the API because it was making things crash. The "indexing" time was too long. I have still check everything was ok in the logs : I saw this : Serving monocular at http://[::]:8081

So I decide to connect into the monocular-api container and try a curl on http://localhost:8081/api/v1/charts and I also have a 404 error ...

Do I missed something?

Thank you for your help.

@prydonius

This comment has been minimized.

Member

prydonius commented Feb 12, 2018

Hey @cabrinoob!

How are you accessing the monocular UI? You need to access it through an ingress server (nginx-ingress controller is recommended) otherwise there is no way for the UI to route to the API.

The ingress strips off the /api prefix, so what you really want to try is curling http://localhost:8081/v1/charts inside your monocular API container.

@cabrinoob

This comment has been minimized.

cabrinoob commented Feb 13, 2018

Hi @prydonius ,
Yes, i have installed the nginx ingress controller as explained in the documentation. My cluster is in AWS and was installed with KOPS. So the ingress created an ELB and I access to the UI through it.

I cannot see the path stripping you talked about in the ingress rule :

{
	"kind": "Ingress",
	"apiVersion": "extensions/v1beta1",
	"metadata": {
		"name": "hipster-peacock-monocular",
		"namespace": "default",
		"selfLink": "/apis/extensions/v1beta1/namespaces/default/ingresses/hipster-peacock-monocular",
		"uid": "f64b4d9f-1009-11e8-a4a3-0207fdbaed84",
		"resourceVersion": "19901707",
		"generation": 1,
		"creationTimestamp": "2018-02-12T15:32:40Z",
		"labels": {
			"app": "hipster-peacock-monocular",
			"chart": "monocular-0.4.9",
			"heritage": "Tiller",
			"release": "hipster-peacock"
		},
		"annotations": {
			"ingress.kubernetes.io/rewrite-target": "/",
			"kubernetes.io/ingress.class": "nginx"
		}
	},
	"spec": {
		"rules": [
			{
				"http": {
					"paths": [
						{
							"path": "/",
							"backend": {
								"serviceName": "hipster-peacock-monocular-ui",
								"servicePort": 80
							}
						},
						{
							"path": "/api/",
							"backend": {
								"serviceName": "hipster-peacock-monocular-api",
								"servicePort": 80
							}
						}
					]
				}
			}
		]
	},
	"status": {
		"loadBalancer": {
			"ingress": [
				{}
			]
		}
	}
}

Then, I tried to curl http://localhost:8081/v1/charts and it worked .... So the pb seems to come from the ingress rule ?

Thank you for your help

@seagetch

This comment has been minimized.

seagetch commented Feb 14, 2018

I had a same problem.
ingress-nginx changed the prefix of annotations since 0.9-beta.18
https://github.com/kubernetes/ingress-nginx/blob/master/Changelog.md
monocular uses old style annotation prefix.
Maybe template should be fixed to adapt for this change.
Workaround for this is to override the annotations as follows.

cat > custom.yaml <<EOF
ingress:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
EOF
helm install monocular/monocular --name monocular -f custom.yaml
@prydonius

This comment has been minimized.

Member

prydonius commented Feb 14, 2018

@seagetch oh wow, good catch! Thanks for that, I'd be happy to accept a PR to update the chart!

@Muahamed

This comment has been minimized.

Muahamed commented Feb 14, 2018

@seagetch ,,Awesome its works for me .

@jonasdkhansen

This comment has been minimized.

jonasdkhansen commented Mar 6, 2018

I have the same errors, but also others like.
@cabrinoob how do you exec into monoculars api container ? I've tried to do a
kubectl exec monocular-api-754ff569bf-8bnhv -c monocular -- /bin/bash
but i'm getting this error

rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "exec: \"/bin/bash\": stat /bin/bash: no such file or directory"

command terminated with exit code 126

@seagetch i've installed monocular with a custom config, and changed the ingress annotation to
nginx.ingress.kubernetes.io/rewrite-target: /as you described, but with no luck.

If i go the the external ip of the nginx ingress controller, then the monocular UI loads as it should, but in the console i'm getting the same errors like:

GET https://NGINX_INGRESS_EXTERNAL_IP/api/v1/charts 404 ()
404 - OK
ERROR 404 - OK
GET https://NGINX_INGRESS_EXTERNAL_IP/api/auth/verify 404 ()

Anybody here that has a suggestion on how i could fix this issue ?

@cabrinoob

This comment has been minimized.

cabrinoob commented Mar 6, 2018

@jonasdkhansen do you mean :

kubectl exec monocular-api-754ff569bf-8bnhv -n monocular -- /bin/bash

with -n (not -c). So yes, it should work. Try with the another api container (who knows)

Regarding 404 http errors, I've got the same in the chrome console, but the UI works, so ....

@jonasdkhansen

This comment has been minimized.

jonasdkhansen commented Mar 6, 2018

@cabrinoob thanks for the answer, i have tried with the command you suggest, and still getting the same error bin/bash: no such file or directory. Everything is green and running in kubernetes.

The UI is also working here, but no charts is loading, it's just spinning. I have checked the log on the api container, and i'm sometimes getting an error like "Local cache missing".

@cabrinoob

This comment has been minimized.

cabrinoob commented Mar 7, 2018

@jonasdkhansen : for the "exec" command, you can try /bin/sh instead /bin/bash

Regarding the spinner, I had the same problem. It was a problem due to the indexing frequency of our S3 repository. Monocular regenerate cache every hour by default. But you can change this value in the monocular-api configmap. You have to change the cacheRefreshInterval value(I updated it from 3600 to 300).

Another possible mistake you could do (and i've done ^^), is to miss including a README.md file in each chart I published on my private s3 repository. I had an error in the monocular-api pod logs.

But passing each of these problems, everything works fine today.

@jonasdkhansen

This comment has been minimized.

jonasdkhansen commented Mar 7, 2018

@cabrinoob if i try with /bin/sh it's just stalling, no errors or anything.

I have changed the frequency to update cache every 300 seconds, and i have not added a custom repository, only the officials:

      - name: stable
        url: https://kubernetes-charts.storage.googleapis.com
        source: https://github.com/kubernetes/charts/tree/master/stable
      - name: incubator
        url: https://kubernetes-charts-incubator.storage.googleapis.com
        source: https://github.com/kubernetes/charts/tree/master/incubator

And do you know what the error "Local cache missing" does mean ?

@cabrinoob

This comment has been minimized.

cabrinoob commented Mar 7, 2018

Are you sure it's staling? Because with sh you just have an # as prompt. I tried it again and /bin/bash does not work, but /bin/sh works for me.

with /bin/sh

Even if you are using official repositories, monocular creates a local cache (I think, I'am not a contributor to this project).

So, try again with /bin/sh (normally it will work), and try to curl https://kubernetes-charts.storage.googleapis.com from inside the container (to be sure your pod can access to internet).

@jonasdkhansen

This comment has been minimized.

jonasdkhansen commented Mar 7, 2018

@cabrinoob Thanks for the help, and the quick responses!
It works with /bin/sh to the container now, and i think i have loaded the wrong config file for installing monocular, so i did not get your configuration change with "nginx.ingress.kubernetes.io/rewrite-target: /" in my install, so it all works now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment