Authentication support for ELK Logging #2300
Comments
|
To be %100 sure we can send logs to Elastic Search i made a simple put which worked fine: |
|
Hey @omerfsen, If you check out the request logging microservice, it accepts a seldon-core/components/seldon-request-logger/app/log_helper.py Lines 142 to 143 in a2f3c25 You can see an example of how it can be used here: https://github.com/SeldonIO/seldon-core/tree/master/examples/centralised-logging On that one in particular, it uses KNative to scale the request logging service but I think it's also possible to just have it there as a stand-alone service. It would be great if you could check out that example to see if it solves your use case. |
|
Yeah the non-knative service is https://github.com/SeldonIO/seldon-core/blob/master/examples/centralised-logging/seldon-request-logger-k8s.yaml. That is still using knative eventing for the trigger though. To cut that out too you'd remove the trigger and change https://github.com/SeldonIO/seldon-core/blob/master/helm-charts/seldon-core-operator/values.yaml#L54 to point at |
|
Oh I think I see the misunderstanding now. |
|
Thank you for the update @ryandawsonuk and @adriangonz i will test this and afterwards if all goes well will write a document on Seldon Centralized Logging to ELK |
|
I have connected to ELK with seldon-request-logger: ( I am using latest version 1.2.3 for seldon-request-logger image) . This requires a seldon-logs namespace to be created before and following will create a service to be used with http://seldon-request-logger.seldon-logs (ie. http://<service>.<namespace>) apiVersion: apps/v1
kind: Deployment
metadata:
name: seldon-request-logger
namespace: seldon-logs
labels:
app: seldon-request-logger
spec:
replicas: 1
selector:
matchLabels:
app: seldon-request-logger
template:
metadata:
labels:
app: seldon-request-logger
spec:
containers:
- name: user-container
image: docker.io/seldonio/seldon-request-logger:1.2.3
imagePullPolicy: Always
env:
- name: ELASTICSEARCH_PROTOCOL
value: https
- name: ELASTICSEARCH_HOST
value: "dfb2795d481dbe235733219ca518.eu-central-1.aws.cloud.es.io"
- name: ELASTICSEARCH_PORT
value: "9243"
- name: ELASTICSEARCH_USER
value: elastic
- name: ELASTICSEARCH_PASS
value: <PASS>
---
apiVersion: v1
kind: Service
metadata:
name: seldon-request-logger
namespace: seldon-logs
spec:
selector:
app: seldon-request-logger
ports:
- protocol: TCP
port: 80
targetPort: 8080
logs: |
|
Wrote a sample deployment: apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
name: iris-model-logging
namespace: seldon-deploy
spec:
name: iris
predictors:
- graph:
implementation: SKLEARN_SERVER
modelUri: gs://seldon-models/sklearn/iris
name: classifier
logger:
url: http://seldon-request-logger.seldon-logs
mode: all
name: default
replicas: 1
svcOrchSpec:
env:
- name: SELDON_LOG_LEVEL
value: DEBUG and after that i can see logs are actually (as mode: all it both sends and respose) kubectl logs -f -n seldon-logs seldon-request-logger-f988d4478-hvrbp |
|
For installation you can also set default request-logger url like: helm upgrade --install seldon-core seldon-core-operator --repo https://storage.googleapis.com/seldon-charts --set istio.enabled=true --set usageMetrics.enabled=false --namespace seldon-system --set executor.requestLogger.defaultEndpoint="http://seldon-request-logger.seldon-logs"and on your manifest apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
name: iris-model-logging
namespace: seldon-deploy
spec:
name: iris
predictors:
- graph:
implementation: SKLEARN_SERVER
modelUri: gs://seldon-models/sklearn/iris
name: classifier
logger:
mode: all
name: default
replicas: 1
svcOrchSpec:
env:
- name: SELDON_LOG_LEVEL
value: DEBUGOR on helm you dont define --set executor.requestLogger.defaultEndpoint="http://seldon-request-logger.seldon-logs" and define logger.url on your yaml like: apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
name: iris-model-logging
namespace: seldon-deploy
spec:
name: iris
predictors:
- graph:
implementation: SKLEARN_SERVER
modelUri: gs://seldon-models/sklearn/iris
name: classifier
logger:
url: http://seldon-request-logger.seldon-logs
mode: all
name: default
replicas: 1
svcOrchSpec:
env:
- name: SELDON_LOG_LEVEL
value: DEBUG
|
|
I have made Env variables to come from a secret so kubeseal or other secret mechanisms works to store secrets on github etc.. So also wrote a small secret generator script #!/bin/bash
if [ -z "${1}" ]
then
echo "provide seldon-logs"
exit 2
else
SELDON_LOGGER=${1}
fi
#########
# YOU CAN USE $(aws kms....) or $(az keyvault secrets) . Below secrets are just for example.
ELASTICSEARCH_PROTOCOL=https
ELASTICSEARCH_HOST=df65d4b2795d4835733219ca518.eu-central-1.aws.cloud.es.io
ELASTICSEARCH_PORT=9243
ELASTICSEARCH_USER=elastic
ELASTICSEARCH_PASS=<PASS>
# USE TOKEN instead of USER/PASS but we generate it anyway with a NULL value
ELASTICSEARCH_TOKEN=NULL
#########
BASE64_ELASTICSEARCH_PROTOCOL=$(echo -n "${ELASTICSEARCH_PROTOCOL}" | base64 -w 0)
BASE64_ELASTICSEARCH_HOST=$(echo -n "${ELASTICSEARCH_HOST}" | base64 -w 0)
BASE64_ELASTICSEARCH_PORT=$(echo -n "${ELASTICSEARCH_PORT}" | base64 -w 0)
BASE64_ELASTICSEARCH_USER=$(echo -n "${ELASTICSEARCH_USER}" | base64 -w 0)
BASE64_ELASTICSEARCH_PASS=$(echo -n "${ELASTICSEARCH_PASS}" | base64 -w 0)
#Either USER/PASS or TOKEN is used
BASE64_ELASTICSEARCH_TOKEN=$(echo -n "${ELASTICSEARCH_TOKEN}" | base64 -w 0)
rm -f ${SELDON_LOGGER}_secrets.yml
cat << __EOF__ > ${SELDON_LOGGER}_secrets.yml
---
apiVersion: v1
kind: Namespace
metadata:
name: ${SELDON_LOGGER}
---
apiVersion: v1
kind: Secret
metadata:
name: ${SELDON_LOGGER}
namespace: ${SELDON_LOGGER}
type: Opaque
data:
ELASTICSEARCH_PROTOCOL: ${BASE64_ELASTICSEARCH_PROTOCOL}
ELASTICSEARCH_HOST: ${BASE64_ELASTICSEARCH_HOST}
ELASTICSEARCH_PORT: ${BASE64_ELASTICSEARCH_PORT}
ELASTICSEARCH_USER: ${BASE64_ELASTICSEARCH_USER}
ELASTICSEARCH_PASS: ${BASE64_ELASTICSEARCH_PASS}
ELASTICSEARCH_TOKEN: ${BASE64_ELASTICSEARCH_TOKEN}
__EOF__
kubectl apply -f ${SELDON_LOGGER}_secrets.yml
and then deployment is like: apiVersion: apps/v1
kind: Deployment
metadata:
name: seldon-request-logger
namespace: seldon-logs
labels:
app: seldon-request-logger
spec:
replicas: 1
selector:
matchLabels:
app: seldon-request-logger
template:
metadata:
labels:
app: seldon-request-logger
spec:
containers:
- name: user-container
image: docker.io/seldonio/seldon-request-logger:1.2.3
imagePullPolicy: Always
env:
- name: ELASTICSEARCH_PROTOCOL
valueFrom:
secretKeyRef:
name: seldon-logs
key: ELASTICSEARCH_PROTOCOL
- name: ELASTICSEARCH_HOST
valueFrom:
secretKeyRef:
name: seldon-logs
key: ELASTICSEARCH_HOST
- name: ELASTICSEARCH_PORT
valueFrom:
secretKeyRef:
name: seldon-logs
key: ELASTICSEARCH_PORT
- name: ELASTICSEARCH_USER
valueFrom:
secretKeyRef:
name: seldon-logs
key: ELASTICSEARCH_USER
- name: ELASTICSEARCH_PASS
valueFrom:
secretKeyRef:
name: seldon-logs
key: ELASTICSEARCH_PASS
# if we use TOKEN define it here. Instead we use USER/PASS
# - name: ELASTICSEARCH_TOKEN
# valueFrom:
# secretKeyRef:
# name: seldon-logs
# key: ELASTICSEARCH_TOKEN
---
apiVersion: v1
kind: Service
metadata:
name: seldon-request-logger
namespace: seldon-logs
spec:
selector:
app: seldon-request-logger
ports:
- protocol: TCP
port: 80
targetPort: 8080 |
|
@omerfsen Can we close this or is there a further change you'd like to see? |
|
Yes we can close it. |
Our ELK endpoints are authenticated so I used a sample sdp like
made a few curl post to Seldon API
but
curl -u user:Pass "https://df65d4b2795d481dbe235733219ca518.eu-central-1.aws.cloud.es.io:9243/_cat/indices"
does not return anything for seldon so requesting authentication support for ELK Logging.
User/Pass can be taken from a Kubernetes Secret.
PS: I also tried to use helm method to install
both did not worked
The text was updated successfully, but these errors were encountered: