Improve Custom/On-Premise Docs

[dwhitena]

There's been a lot of interest in on-prem clusters lately, and our docs on the subject aren't very good. The main issues that people have been hitting are:

• When to use the custom deploy
• Confusion around how to use the custom deploy for cloud vs. on prem deploys
• How to modify the manifest for on prem deploys
• What needs to be changed in the deploy process for OpenShift, OpenStack, and other systems.
• NotExist error with CEPH S3 interface deploy

I think the following will probably make the process better for on prem users:

• Use the Helm chart as default for on prem deploys. Our users seem to indicate in Slack that this is much easier.
• Update the custom/on-prem docs to emphasize custom object stores in the cloud vs. completely on-prem solutions.
• Test the custom deploy commands and the Helm chart to see if updates are needed for the latest Pachyderm versions.

[naztyroc]

Hi Dan,

We were attempting to deploy an instance of pachyderm with our CEPH S3 gateway protocol as a replacement for a MINIO deployment, but ran into the error you explained in the intro of this issues report. This forced us to deploy using the helm chart provided in the official kubernetes charts repo. This was a GREAT help! To make things even better for our use case, we were now able to deploy multiple instances of pachyderm for creating serval pipelines using different storage backends in the same K8 cluster. We encountered a few problems along the way which required us to modify the helm chart pr 4159 This enabled us to deploy multiple instances of pachyderm using CEPH S3 gateway, MINIO, Amazon, etc.. Many thanks!!

[dwhitena]

Thanks for you comments and contribution @naztyroc! This is great context for us as we improve the on prem docs. I think Jon Ander is reviewing your Helm chart PR, and we will move forward on the docs side here.

[liuchenxjtu]

hi Dan,
thanks your for your efforts. I am quite new to Pachyderm. now we want to deploy pachyderm on our own cluster. but most of the docs are about deploying on cloud provider, e.g., persistent-disk must be google, azure or aws. so for our case, could you advise what we should be for the local clusters? thank you!

[naztyroc]

Hi Dan, For our use case, we are using CEPH as the storage backend for Kubernetes which connects as RBD volumes. We configured Kubernetes to communicate with CEPH by creating storage class methods. With that setup, we deployed Minio, with CEPH/RBD PVC, as the backend for Pachyderm which talks to it over S3 protocol. I hope that helps?

…

-Rashian

Sent from my iPhone

On Apr 26, 2018, at 9:11 PM, liuchenxjtu ***@***.***> wrote: hi Dan, thanks your for your efforts. I am quite new to Pachyderm. now we want to deploy pachyderm on our own cluster. but most of the docs are about deploying on cloud provider, e.g., persistent-disk must be google, azure or aws. so for our case, could you advise what we should be for the local clusters? thank you! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[dwhitena]

Thanks for the update @naztyroc. Will hopefully have these updated soon.

@liuchenxjtu Thanks for reaching out. I would recommend using the Helm chart for your deploy: https://github.com/kubernetes/charts/tree/master/stable/pachyderm. You can use Minio, Rook, Ceph, Swift, etc. for your object store, and then use can use can any k8s PV for the volume: https://kubernetes.io/docs/concepts/storage/persistent-volumes/

[liuchenxjtu]

Hi Dan,
thanks for your reply. yeah I am working on that. but how can I deploy pachyderm on my local kubernete cluster? now it seems it put the data in a local folder, how should I configure to put it in Minio? sorry could not find many documents

[JoeyZwicker]

@liuchenxjtu a local deployment (minikube) of Pachyderm just uses local storage and not a full object store. It's also not really meant for production workloads where you'd need a real object store.

If you want to use minio as the storage backend for a tiny cluster, you'll need to do a single-node or on-prem deployment of kubernetes and then use the custom deploy or helm chart as described above.

If you tell me a bit more about your setup and requirement, we can give you additional guidance. For example:

could you advise what we should be for the local clusters

When you say "local clusters" do you mean on-prem or do you mean a minikube or "pachctl deploy local" cluster?

[liuchenxjtu]

Hi @dwhitena ,
sorry for the late reply.

It's also not really meant for production workloads where you'd need a real object store.
so this means if I want to make it production level, I need to set up an object store? can I use glusterFS?
When you say "local clusters" do you mean on-prem or do you mean a minikube or "pachctl deploy local" cluster?
our local cluster is a 1 master,3 node kubernetes cluster, we built it on our own servers but not in the cloud. I guess I need the on-prem setting. but according to the document there, the persistent-disk parameter I can only use aws,google or azure. but for my case, I did not use any of them, how should I set? thanks for your help!

[dwhitena]

@liuchenxjtu

Regarding gluster FS, yes you can use gluster FS, which a Minio server as an S3 compatible layer between the two.

Regarding the persistent disk, We are changing out on-prem default to use the Helm chart. That should clear up the confusion around the PVs. More details can be found here: https://github.com/kubernetes/charts/tree/master/stable/pachyderm

[liuchenxjtu]

hi @dwhitena , @JoeyZwicker
thanks for your reply. sorry for my poor knowledge. could you clarify whether I can build pachyderm directly on glusterFS?
in addition, I tried to set up Pachyderm with Minio by following command helm install --name my-release1 --set credentials=s3,s3.accessKey=AKIAIOSFODNN7EXAMPLE,s3.secretKey=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY,s3.bucketName=test,s3.endpoint="http://172.16.41.154:9000",etcd.persistence.enabled=false,etcd.persistence.accessMode=ReadWriteMany pachyderm

when I tried to put file in one repo, e.g., pachctl put-file test master -c -f lsof.log
"The specified bucket does not exist". if I ran it again, it shows "rpc error: code = Unknown desc = parent commit 92d19698780b456d9f4fdeb734350e1e has not been finished". but I do have the bucket in minio. it is also set up by helm charts. there is log from pod.
2018-05-08T09:13:36Z ERROR pfs.API.GetFile {"duration":0.004858689,"error":"objectInfoGetter: object c79f09e533b8de46b88eea1267f97842ed6e4d93104d8d4879306d1561bbb48a1b580f0bc2adeb20af24f6d2d4facb4e12c1ef3f6825cdd86cde0bae363d292c not found","request":{"file":{"commit":{"repo":{"name":"spec"},"id":"f2541bff66c64ac48fa3541d4b72f9c4"},"path":"spec"}}}
time="2018-05-08T09:13:36Z" level=error msg="master: error running the master process: watch event had no pipelineInfo: could not read existing PipelineInfo from PFS: objectInfoGetter: object c79f09e533b8de46b88eea1267f97842ed6e4d93104d8d4879306d1561bbb48a1b580f0bc2adeb20af24f6d2d4facb4e12c1ef3f6825cdd86cde0bae363d292c not found; retrying in 12.919605042s"

thanks for your help!!

[marcadella]

I have the same issue: I'd like to use pachctl deploy local but with Minio instead of what local uses (probably some kind of local directory). The issue is that custom expects us to use provide --persistent-disk (with for only options google, aws or azure).

[brokenjacobs]

Will the helm chart be updated to 1.8?
Can you configure replicas for etcd with the helm chart?
How do you deploy the dashboard with the helm chart?

[gabrielgrant]

@brokenjacobs the helm chart is still not (yet) an officially/fully supported deployment method for Pachyderm, as it hasn't been incorporated into the test suite or build/release process. That being said, it would be great if you could file each of those three questions as it's own issue. Hopefully someone on the pachyderm core team or a community member can get around to answering/adding each of those things soon.

[gabrielgrant]

@marcadella most likely what you'll want to do is run pachctl deploy custom --dry-run [YOUR_OPTIONS] > manifest.json using options that are as close as possible to what you want for your deployment. This will output a Kubernetes manifest into manifest.json that you can then edit to suit your particular environment.

[brokenjacobs]

I was able to get it deployed using the custom option. I was scared off by the persistent disk option being required but it didn’t seem to put anything into the output manifest.

It’s not working properly though, dashboard won’t come up. I suspect k8s 13.2 may be an issue. Will investigate on Monday.

[iswaverly]

Hi Dan,

We were attempting to deploy an instance of pachyderm with our CEPH S3 gateway protocol as a replacement for a MINIO deployment, but ran into the error you explained in the intro of this issues report. This forced us to deploy using the helm chart provided in the official kubernetes charts repo. This was a GREAT help! To make things even better for our use case, we were now able to deploy multiple instances of pachyderm for creating serval pipelines using different storage backends in the same K8 cluster. We encountered a few problems along the way which required us to modify the helm chart pr 4159 This enabled us to deploy multiple instances of pachyderm using CEPH S3 gateway, MINIO, Amazon, etc.. Many thanks!!

Hi, I am trying to use CEPH by s3 gateway. How can I config charts? I set s3 configuration in values.yaml, and It seems to use Minio as backend storage. It would be of great help if you could show me how you config charts.

[gabrielgrant]

@iswaverly Sorry for the confusion, Pachyderm uses the Minio client to communicate with all non-Amazon S3-compatible object stores (including Ceph). So it sounds like you've got it set up correctly

[pappasilenus]

@JoeyZwicker @svekars Do you think we could close this, with the work done on custom deployment last year?

[svekars]

Sounds good to me!

[mindthevirt]

Closing as this has been addressed by JK.