NFS: feature: NFS Export management API

[BlaineEXE]

I would like to request a new go-ceph API to manage NFS exports. For now, I will start with a loose overview of general needs for the API without specifically defining API functions:

• List NFS exports belonging to a given NFS cluster
• Get detailed information about an NFS export belonging to a given NFS cluster
• Create an export for a given NFS cluster
• Update an export for a given NFS cluster
• Delete an NFS export belonging to a given NFS cluster
• (optional) List NFS clusters; Rook will already know the name of the NFS cluster and so doesn't need to list them, but this may be good to include for completion. Since this is about NFS clusters and not Exports, it may mean the API should be general NFS management and not specifically export management. Export management may be a sub-API depending on how go-ceph feels it is best to design the API.

There are NFS manager module commands that can manage exports via CLI. This has been my previous experience managing NFS.
I believe the go-ceph project has opted not to use Ceph CLI commands, so I assume the CLI is off the table. If that isn't the case and the CLI ends up being preferred, there is an additional hang-up: the nfs manager module CLI requires an orchestrator backend be enabled. We don't always want to enable this, and so having the option to add flags to the nfs commands to run in orchestrator-less configuration would be ideal.

I also see documentation regarding an NFS export REST API. I have used the ceph-dashboard to manage NFS exports, but I am not sure if the dashboard uses these API endpoints or uses the shared NFS Python code directly. From initial appearances, this API meets the general needs outlined above. It is unclear to me whether the API is up-to-date with regards to using the same Python code path as the dashboard and nfs module CLI; if not, it may not automatically create a Ceph user for the export.

Ideally, I believe go-ceph would use the REST API to manage exports to make use of existing work done in Ceph. If neither the REST API nor NFS mgr module CLI are good choices for go-ceph, manually managing exports can be done and is not terribly difficult. When creating an export:

1. a Ceph user must be created for the export
2. a namespaced rados object containing nfs-ganesha export configuration information (example here) must be created
3. the namespaced rados config object for nfs-ganesha must be modified to reference the created export config object

Deleting an export happens in the reverse order, and updates must merely update the export config object via rados.

[phlogistonjohn]

Typically, what we've been doing is to perform the same API calls that the ceph cli tool does but without forking/exec'ing the cli. In other words we use the same mon_command & mgr_command APIs with JSON requests and responses - just like ceph CLI does under the covers.

Currently, the only parts of go-ceph that make use of HTTP APIs are related to rgw, but if there is a compelling reason to use them we could. But we'd need to determine that the HTTP API is somehow better for our use case(s) than the CLI-equivalent json command apis.

So, in short I think that we can add the needed apis for automating these in go. Perhaps the next step would be listing ceph nfs ... command lines you'd like to have an equivalent of?

[BlaineEXE]

Good to know! As for the nfs CLI commands, the important ones are:

• create: https://docs.ceph.com/en/latest/mgr/nfs/#create-cephfs-export
• delete: https://docs.ceph.com/en/latest/mgr/nfs/?#delete-export
• list: https://docs.ceph.com/en/latest/mgr/nfs/?#list-exports
• get: https://docs.ceph.com/en/latest/mgr/nfs/?#get-export

Critically, the NFS CLI requires that the Rook orchestrator module be enabled, which we want to avoid. If we are using these commands, we (Rook) really want to see the internal Python stuff reworked such that they avoid checking the passed arguments with the Rook orchestrator module (which I believe is only a check to see if the NFS "cluster" exists in Rook). This can be an optional "--no-orchestrator" flag to the CL, or it could be changing the internals to use a different base assumption.

I wonder if we should chat with Juan Mi about this part. Are there other mgr/Python people?

[phlogistonjohn]

https://tracker.ceph.com/issues/54043 -- I filed a ceph tracker issue after I briefly discussed this with Sebastian W, @adk3798, and others in a ceph orchestration meeting.

[BlaineEXE]

Cross-linking to related Ceph PR here for my own tracking: ceph/ceph#44870

[nixpanic]

Note that ceph/ceph-csi#2948 is an initial Ceph-CSI provisioner for NFS. It currently calls out to the ceph command, and will use the go-ceph implementation when ready.

[phlogistonjohn]

@nixpanic the PR is blocked waiting on backports in ceph (ceph/ceph#45508). However, these changes are needed only to run without any orchestration module. If the equivalent ceph cli commands are working for your test cases now, then you should be able to start trying out the go-ceph APIs if you wish. Ultimately, the cli commands and the go-ceph functions are using the same "endpoints" in the ceph mgr module.