New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items' #78

Open
lejoow opened this Issue May 9, 2018 · 21 comments

Comments

Projects
None yet
@lejoow
Copy link

lejoow commented May 9, 2018

I am not 100% sure if this is the issue specifics to my local settings or not. But every time I try to deploy the item-svc or bidding-svc to my Google KBE, I am getting this exception.

Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'

screen shot 2018-05-09 at 7 23 54 am

@TimMoore

This comment has been minimized.

Copy link
Member

TimMoore commented May 10, 2018

I think I've found the problem. GCP enables Role-Based Access Control (aka RBAC) by default in new Kubernetes clusters. This requires configuring roles to allow access to resources within the Kubernetes API. Lightbend Orchestration (specifically, the Akka Cluster Bootstrap component) uses the Kubernetes API to automatically form a cluster. Without binding your pods' service account to a role that grants access to the required resources, these API calls fail.

To correct this, you'll need to follow the instructions on this page for configuring role-based access control: https://developer.lightbend.com/docs/akka-management/current/discovery.html#discovery-method-kubernetes-api

For example, you can supply these Kubernetes resources to kubectl apply:

---
#
# Create a role, `pod-reader`, that can list pods and
# bind the default service account in the `default` namespace
# to that role.
#

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: read-pods
subjects:
# Note the `name` line below. The first default refers to the namespace. The second refers to the service account name.
# For instance, `name: system:serviceaccount:myns:default` would refer to the default service account in namespace `myns`
- kind: User
  name: system:serviceaccount:default:default
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Once you've created this role and role binding, your Lagom services should start correctly. You might want to tailor this to your specific security needs.

@lejoow please let us know if that fixes the problem for you. Even if it does, we can leave this issue open until we update the documentation.

@lejoow

This comment has been minimized.

Copy link

lejoow commented May 10, 2018

@TimMoore fantastic. Let me try this fix today and come back to you on this post. Thanks a lot!

@lejoow

This comment has been minimized.

Copy link

lejoow commented May 10, 2018

hmm... I don't get that error anymore, but I am getting the different errors now:

2018-05-10T17:54:27.363Z [�[37minfo�[0m] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=application-akka.actor.default-dispatcher-24, akkaTimestamp=17:54:27.363UTC, akkaSource=akka.tcp://application@10.16.1.48:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=application] - Exceeded stable margins without locating seed-nodes, however this node is NOT the lowest address out of the discovered IPs in this deployment, thus NOT joining self. Expecting node Some(ResolvedTarget(10.16.0.13,Some(10002))) (out of List(ResolvedTarget(10.16.0.13,Some(10002)), ResolvedTarget(10.16.1.48,Some(10002)))) to perform the self-join and initiate the cluster.

I am still investigating..

@lejoow

This comment has been minimized.

Copy link

lejoow commented May 10, 2018

It worked! I just had to re-deploy. Thanks!
Btw, when I tried to create the role by applying that file you pasted, I ran into "pod-reader forbidden" error message, which I resolved by following the instructions here:

kubectl create clusterrolebinding cluster-admin-binding
--clusterrole=cluster-admin
--user=$primary_account
https://github.com/jcbsmpsn/gke-rbac-walkthrough/blob/master/create-roles.md

@lejoow lejoow closed this May 10, 2018

@TimMoore TimMoore reopened this May 11, 2018

@TimMoore

This comment has been minimized.

Copy link
Member

TimMoore commented May 11, 2018

Let's leave this open until the documentation is updated. Thanks!

@TimMoore

This comment has been minimized.

Copy link
Member

TimMoore commented May 15, 2018

Added an issue for the Akka Management documentation to note that Google Cloud users will need to grant themselves the cluster-admin ClusterRole: akka/akka-management#199

@Canislupax

This comment has been minimized.

Copy link

Canislupax commented May 20, 2018

That solved my problem! Thanx a lot Tim! You`re the best!

@idoshamun

This comment has been minimized.

Copy link

idoshamun commented May 27, 2018

@TimMoore I followed your suggestion but now I receive:

14:47:46.025 [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'ports'

For what it's worth, I am trying to run Lagom with Istio

@idoshamun

This comment has been minimized.

Copy link

idoshamun commented May 27, 2018

Disabling Istio resolves this issue, I will try to look further and share results

@normenmueller

This comment has been minimized.

Copy link

normenmueller commented May 31, 2018

@TimMoore @lejoow I have the same issue running on Minikube. Here are my deployment steps:

$ sbt docker:publishLocal

$ minikube delete
$ minikube start --memory 8192
$ eval $(minikube docker-env)          <<< Should this command come before docker:publishLocal?
$ minikube addons enable ingress
$ kubectl apply -f RBAC.yml            <<< The file posted by @TimMoore

$ sbt "deploy minikube"

$ kubectl cluster-info
$ kubectl get deployments
$ kubectl get pods
$ echo "http://$(minikube ip)"

All Pods have status Running.

Then I had a look at the minikube dashboard and all my services are marked with

Readiness probe failed: Get http://172.17.0.8:10002/platform-tooling/ready: dial tcp 172.17.0.8:10002: getsockopt: connection refused

Then I had a look in the logs at the minikube dashboard and get:

[error] akka.actor.ActorSystemImpl [sourceThread=application-akka.actor.default-dispatcher-2, akkaTimestamp=16:32:22.971UTC, akkaSource=akka.actor.ActorSystemImpl(application), sourceActorSystem=application] - Failed to unmarshal Kubernetes API response status [400 Bad Request]; check RBAC settings
@Bunyod

This comment has been minimized.

Copy link

Bunyod commented May 31, 2018

@TimMoore @lejoow @normenmueller I'm also facing with the same getsockopt: connection refused issue for Minikube. I follow this link.

Does anybody have an idea how to fix it?

@OguzhanSaltik

This comment has been minimized.

Copy link

OguzhanSaltik commented Jun 20, 2018

We have the same error, due to Lightbend Orchestration documentation.

@TimMoore

This comment has been minimized.

Copy link
Member

TimMoore commented Jun 20, 2018

The getsockopt: connection refused message itself only indicates that the service is not running. Of course, there could be many reasons for that. You'll need to look at the logs for the pod to understand more. I'd like to keep this issue thread focused on the original issue reported, which has a solution described in a comment above: #78 (comment)

If you're having another problem, the best place to start is by posting an issue on the Discourse forum: https://discuss.lightbend.com/

@bahtiyarerden

This comment has been minimized.

Copy link

bahtiyarerden commented Jun 21, 2018

We are trying to deploy our lagom application with kubernetes. But we could not fix this issue. We clone this repository (chirper) and deploy with minikube to be sure the problem does not about our configuration. It is the result:

screen shot 2018-06-21 at 15 29 29

And this is the log file:

2018-06-21T12:36:25.703Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-21, akkaTimestamp=12:36:25.703UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:26.756Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-15, akkaTimestamp=12:36:26.756UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:26.883Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-21, akkaTimestamp=12:36:26.883UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:27.802Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:27.801UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:28.839Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:28.839UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:28.882Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-14, akkaTimestamp=12:36:28.882UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:29.863Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-14, akkaTimestamp=12:36:29.863UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:30.883Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-4, akkaTimestamp=12:36:30.882UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:30.936Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-4, akkaTimestamp=12:36:30.935UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:32.051Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:32.051UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:32.893Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:32.892UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:33.089Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:33.088UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:34.116Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:34.115UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:34.873Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:34.872UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:35.159Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:35.157UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:36.257Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-15, akkaTimestamp=12:36:36.257UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:36.873Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:36.873UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:37.304Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-20, akkaTimestamp=12:36:37.301UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:38.332Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:38.332UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:38.885Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-2, akkaTimestamp=12:36:38.885UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:39.378Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:39.376UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:40.420Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-18, akkaTimestamp=12:36:40.420UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'
2018-06-21T12:36:40.882Z [warn] akka.cluster.sharding.ShardRegion [sourceThread=chirpservice-akka.actor.default-dispatcher-15, akkaTimestamp=12:36:40.882UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/sharding/ChirpRepositoryImpl%24ChirpTimelineEventReadSideProcessor, sourceActorSystem=chirpservice] - No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [3] buffered messages.
2018-06-21T12:36:41.468Z [warn] akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=chirpservice-akka.actor.default-dispatcher-21, akkaTimestamp=12:36:41.468UTC, akkaSource=akka.tcp://chirpservice@172.17.0.8:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=chirpservice] - Resolve attempt failed! Cause: spray.json.DeserializationException: Object is missing required member 'items'

I think the problem is about configuration that given below. My microservices that do not have this configuration are working without any error. When i look the logs then i see the akka management port is 10002. The readiness prob also check this port and getting getsockopt: connection refused error.

The application.conf file configuration:

akka.actor.serialization-bindings {
  "akka.Done"                 = akka-misc
  "akka.actor.Address"        = akka-misc
  "akka.remote.UniqueAddress" = akka-misc
}
@ale222

This comment has been minimized.

Copy link

ale222 commented Jun 23, 2018

Trying to deploy online-auction, getting:

akka.management.cluster.bootstrap.dns.HeadlessServiceDnsBootstrap [sourceThread=application-akka.actor.default-dispatcher-5, akkaTimestamp=12:27:31.874UTC, akkaSource=akka.tcp://application@10.48.0.13:10001/system/headlessServiceDnsBootstrap, sourceActorSystem=application] - Discovered [1] observation, which is less than the required [2], retrying (interval: 1.000 s)

any ideas?

@zhangzhonglai

This comment has been minimized.

Copy link

zhangzhonglai commented Jun 27, 2018

same error because of akka/akka-management#222 when using sbt-reactive-app >= 1.2.0
if someone encounter the error like me, simply back to use sbt-reactive-app 1.1.0 will temporary fix the error now.

@ktoso

This comment has been minimized.

Copy link

ktoso commented Jun 27, 2018

The related issue #222 was resolved, 0.15 should fix it then. Thanks for digging into this!

@OguzhanSaltik

This comment has been minimized.

Copy link

OguzhanSaltik commented Jun 27, 2018

Great news @ktoso , when will be 0.15 version applied to the Lagom?

@ktoso

This comment has been minimized.

Copy link

ktoso commented Jun 27, 2018

I'll have to rely this to @TimMoore, not sure on release schedule of intermediate deps etc.

@TimMoore

This comment has been minimized.

Copy link
Member

TimMoore commented Jun 27, 2018

Lagom has no dependency on akka-management. That comes in by way of Lightbend Orchestration, so I think @mitkus is the right one to ask.

@OguzhanSaltik

This comment has been minimized.

Copy link

OguzhanSaltik commented Jun 28, 2018

You are right, the related fix is part of the sbt-reactive-app plugin. We look forward to the version updates, thanks to everyone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment