connection refused while creating a solr_collection

[aberfeldy]

Describe the bug
On creating a solr collection via this crd:

apiVersion: solr.bloomberg.com/v1beta1
kind: SolrCollection
metadata:
  name: escenic
  namespace: solr
spec:
  solrCloud: escenic
  collection: escenic
  routerName: compositeId
  routerField: id
  autoAddReplicas: true
  numShards: 5
  replicationFactor: 2
  maxShardsPerNode: 1
  collectionConfigName: escenic

I'm getting an error:

{
	"namespace": "solr",
	"cloud": "escenic",
	"collection": "escenic",
	"error": "Recieved bad response code of 400 from solr with response: {\"responseHeader\":{\"status\":400,\"QTime\":3},\"error\":{\"metadata\":[\"error-class\",\"org.apache.solr.common.SolrException\",\"root-error-class\",\"org.apache.solr.common.SolrException\"],\"msg\":\"Collection: escenic not found\",\"code\":400}}\n"
}

This happens because it tries to call solr by the headless service ("$(POD_HOSTNAME)." + solrCloud.HeadlessServiceName()) which is not available in the cluster.
In this case the call to http://escenic-solrcloud-3.escenic-solrcloud-headless:80/solr gets a connection refused, while the call to http://escenic-solrcloud-3.sorl:80/solr (the none-headless service) is completly fine.

To Reproduce
Steps to reproduce the behavior:

1. followed the tutorial in the readme until creation of a solr_collection

Expected behavior
To see the collection be created or to use at least services that exist. Maybe I'm missing some step to create the expected services. Or it should be possible to to use the existing svc (pod-hostnema.namespace) instead, but when I change that in the yaml, it gets immediately reverted to "$(POD_HOSTNAME)." + solrCloud.HeadlessServiceName() (which is good, but doesn't work for me)

Environment (please complete the following information):

• Kubernetes 15.9

[HoustonPutman]

Thanks for filing the bug! Sorry things didn't work for you out of the box.

I'm curious why the headless service doesn't work for you. I just checked because I thought that individual SolrNode Services were only created when an "IngressBaseDomain" is passed in, but it seems like they are always created and only used when an Ingress is requested.

So I guess the steps are to either:

• Always use the individual node services for everything, basically eliminating the need for the headless service
• Stop creating individual node services when ingresses are not requested. And figure out why the headless service isn't working correctly.

Have you been able to use headless services in you kube cluster outside of the solr operator?

[aberfeldy]

The ingress solution works like a charm, I just don't want to ingress them to the outside, since the only ingress controller we use is facing to the public.

Yes the headless services itself work quite nicely, although I don't use them with the pod-name specified, because the headless service points to a DaemonSet. it usually picks the pod on the node the request is from anyway.

Well the quick fix would probably be using the individual services, although I'd be curious why the headless refuses the connection.

If you need additional logging or information let me know it.

[HoustonPutman]

Could you look through the logs and give me all of the context around what you see here? It should log out the action it's trying to take before the error is logged. That will help us figure out the exact issue.

[HoustonPutman]

Are you running istio, or another service mesh? There seems to be issues using headless services with them.

istio/istio#7495
istio/istio#5992

[aberfeldy]

Sorry for the delay.
I was not able to reproduce the error. Cleaning everything up and rebuilding the operator showed no error. The solr cloud is created perfectly without any issues. I wonder why it couldn't connect to the headless before.
We don't have istio installed, there is no special service mesh used.

The only thing I need to adjust, is the memory limit for the operator. It gets OOMkilled constantly before I can even install the cloud.

[HoustonPutman]

Thanks for the update! It would probably be a good idea to increase those defaults.