Spline UI "Server is not available" in the execution events and Data Sources section

[VishweshwarKammari]

I have deployed spline on master node of EMR for a POC. I have followed the process as per the documentation for docker deployment. I don't see any errors in the logs. Able to see spline rest server, spline ui on the respective ports. I have started a pyspark shell using the spark agent jar as below. I can not see any lineage generated in ui. It is displaying server error "Server is unavailable". I am new to spline. Kindly help me fixing this issue.

I am using spline 0.6.0 version with the following process.

wget https://raw.githubusercontent.com/AbsaOSS/spline-getting-started/main/docker/docker-compose.yml

wget https://raw.githubusercontent.com/AbsaOSS/spline-getting-started/main/docker/.env

docker-compose up

pyspark --jars spark-2.4-spline-agent-bundle_2.11-0.6.4.jar --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" --conf "spark.spline.producer.url=http://localhost:8080/producer" --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener"

[wajda]

Open a development console in your browser (F12) and look at the Network tab. You probably have the server IP inaccessible for your browser.

[varankarpranay]

Hi Wajda, I also trying to deploy spline on Kubernetes and Docker VM but getting 404 error for "/consumer" & "/producer" link in browser. attaching docker steps for better understanding.

docker run -d -p 8529:8529 -e ARANGO_NO_AUTH=1 arangodb/arangodb:3.8.1
docker run -it --rm absaoss/spline-admin:0.7.4 db-init arangodb://172.17.0.1:8529/spline
docker container run -d -e SPLINE_DATABASE_CONNECTION_URL=arangodb://172.17.0.1:8529/spline -p 8080:8080 absaoss/spline-rest-server:0.7.4

i have login inside spline docker container and try to hit the url, it give me same 404 error. please refer attached screenshot.

[wajda]

It's correct. There is nothing on the http://172.17.0.1:8080/consumer URL - it's just a base URL for the other Consumer REST API resources.
To check the server status try curl http://172.17.0.1:8080/about/version or curl -I http://localhost:8080/about/liveness
If you still get 404 check Catalina logs in /usr/local/tomcat/logs inside the Spline server container.

[varankarpranay]

i was able to get a response when i hit "/about/version" on spline-rest-server but when i deploy UI getting "server is unavailable" on web browser. i was trying to access UI from internal network. attaching a error for your reference.

[wajda]

Is there any details in the browser developer tool?
I don't know what browser are you using, but usually it's opened by pressing F12.

Look for the Networking tab and failed AJAX requests. Alternatively check the Console tab.

Example of the Google Chrome:

[varankarpranay]

I am using chrome browser. look like i am getting "Cross-Origin Request Blocked" error in console tab and also getting 304 error for some JS in networking tab.

Any idea how to resolve this CORS error, currently my spline deployed on kubernetes cluster on EKS.

[VishweshwarKammari]

Hi @wajda,

Thanks for your inputs and I am able to access UI and see sample lineage. But When I launched a pyspark job with below configuration and tried to run a small job to load sample data to S3. Lineage was not delivered to spline UI and got http 500 error code. I have attached the screenshot below. Please let me know what could be the cause of this issue.

pyspark   --jars spark-3.1-spline-agent-bundle_2.12-0.6.4.jar \
--conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" \
--conf "spark.spline.producer.url=http://internal-k8s-default-splinein-4abb335d06-1548973854.us-west-2.elb.amazonaws.com:8081/producer" 

[wajda]

Interesting, we didn't have issues with CORS before. How did you fix it in the end?

Regarding the server error, look at the server logs (/usr/local/tomcat/logs/ inside the container) for the same UUID that you receive on the client ("errorId" property).

[VishweshwarKammari]

We are using deploying spline on EKS and earlier in our configuration we were using private ips we replaced it with the load balancer and the issue with the communication was resolved.

We looked into the logs of tomcat in the EKS cluster where spline is currently deployed. We have searched in the logs using the error UUID as you suggested. We are using Arangodb 3.8.1 which is latest. Below is the error that we are seeing. Does this sound familiar to you. Appreciate any suggestions.

[wajda]

Oops, looks like you've spotted an oversight. Thank you! 👍
It's a missing collection declaration in the AQL WITH statement.
I've fixed it in the #984
Could you please test the PR and let me know if it works?

(To build the project with the Docker image use mvn install -Ddocker -Ddockerfile.repositoryUrl=my)

[wajda]

The reason why we didn't noticed it is because this statement is optional in a non-cluster ArangoDB setup that is used on our CI and development environments.

[varankarpranay]

Is Spline deployment support cluster ArangoDB setup? also is the reason we get "AQL: collection not known to traversal: 'dataSources'" error because we used arangodb cluster mode in EKS?

FYI,
apiVersion: "database.arangodb.com/v1"
kind: "ArangoDeployment"
metadata:
name: "arangodb-cluster"
spec:
mode: Cluster
image: 'arangodb/arangodb:3.8.1'
environment: Production
externalAccess:
type: NodePort
tls:
caSecretName: None

[wajda]

Yes, that particular error manifests itself on a cluster ArangoDB setup only.
Does #984 fixes the issue for you?

[VishweshwarKammari]

Hi @wajda,

We are using spline 0.6.0 and following docker deployment process. We are not using maven build as we are deploying this in EKS. I am not sure where this update/fix has to be made for the deployment method that we are following. Below is the method we followed for deploying spline 0.6.0. We have used docker-compose.yml file and converted that to a kubernetes yaml file for deploying it in EKS.

Please suggest.

wget https://raw.githubusercontent.com/AbsaOSS/spline-getting-started/main/docker/docker-compose.yml

wget https://raw.githubusercontent.com/AbsaOSS/spline-getting-started/main/docker/.env

[wajda]

Don't worry, I've tested it, the issue is fixed. An updated version (0.7.5) will be available in the public repos tomorrow.

[VishweshwarKammari]

That's a great news. We will start implementing it as soon as it is deployed. We will be waiting to hear from you on this!

[wajda]

Version 0.7.5 is out - https://hub.docker.com/layers/absaoss/spline-rest-server/0.7.5/images/sha256-36386fe1cf93c6b1a9cea613772496f94ed4f2201cb4bc7a0ad163915f09373b?context=explore

[VishweshwarKammari]

Hi @wajda,

Thank you so much for helping us out on this issue. We had deployed spline rest server 0.7.5 and it is working as expected. Thank you for responding quickly to our issues and fixing the bugs.