TestCloneSubVolumeSnapshot failing on ceph main branch

[anoopcs9]

New failures from nightyl CI runs:

=== RUN   TestCloneSubVolumeSnapshot
    clone_test.go:82: 
        	Error Trace:	/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:82
        	Error:      	Received unexpected error:
        	            	API call not implemented server-side: No handler found for 'fs subvolumegroup create'
        	Test:       	TestCloneSubVolumeSnapshot
    clone_test.go:93: 
        	Error Trace:	/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:93
        	Error:      	Received unexpected error:
        	            	API call not implemented server-side: No handler found for 'fs subvolume create'
        	Test:       	TestCloneSubVolumeSnapshot
    clone_test.go:100: 
        	Error Trace:	/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:100
        	Error:      	Received unexpected error:
        	            	API call not implemented server-side: No handler found for 'fs subvolume snapshot create'
        	Test:       	TestCloneSubVolumeSnapshot
    clone_test.go:122: 
        	Error Trace:	/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:122
        	Error:      	Received unexpected error:
        	            	API call not implemented server-side: No handler found for 'fs subvolume snapshot clone'
        	Test:       	TestCloneSubVolumeSnapshot
    clone_test.go:130: 
        	Error Trace:	/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:130
        	Error:      	Received unexpected error:
        	            	API call not implemented server-side: No handler found for 'fs clone status'
        	Test:       	TestCloneSubVolumeSnapshot
    clone_test.go:131: 
        	Error Trace:	/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:131
        	Error:      	Expected value not to be nil.
        	Test:       	TestCloneSubVolumeSnapshot
    clone_test.go:125: 
        	Error Trace:	/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:125
        	            				/opt/go/src/runtime/panic.go:770
        	            				/opt/go/src/runtime/panic.go:261
        	            				/opt/go/src/runtime/signal_unix.go:881
        	            				/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:132
        	Error:      	Received unexpected error:
        	            	API call not implemented server-side: No handler found for 'fs subvolume rm'
        	Test:       	TestCloneSubVolumeSnapshot
    clone_test.go:103: 
        	Error Trace:	/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:103
        	            				/opt/go/src/runtime/panic.go:770
        	            				/opt/go/src/runtime/panic.go:261
        	            				/opt/go/src/runtime/signal_unix.go:881
        	            				/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:132
        	Error:      	Received unexpected error:
        	            	API call not implemented server-side: No handler found for 'fs subvolume snapshot rm'
        	Test:       	TestCloneSubVolumeSnapshot
    clone_test.go:96: 
        	Error Trace:	/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:96
        	            				/opt/go/src/runtime/panic.go:770
        	            				/opt/go/src/runtime/panic.go:261
        	            				/opt/go/src/runtime/signal_unix.go:881
        	            				/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:132
        	Error:      	Received unexpected error:
        	            	API call not implemented server-side: No handler found for 'fs subvolume rm'
        	Test:       	TestCloneSubVolumeSnapshot
    clone_test.go:85: 
        	Error Trace:	/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:85
        	            				/opt/go/src/runtime/panic.go:770
        	            				/opt/go/src/runtime/panic.go:261
        	            				/opt/go/src/runtime/signal_unix.go:881
        	            				/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:132
        	Error:      	Received unexpected error:
        	            	API call not implemented server-side: No handler found for 'fs subvolumegroup rm'
        	Test:       	TestCloneSubVolumeSnapshot
--- FAIL: TestCloneSubVolumeSnapshot (0.15s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x5fbc4b]

goroutine 16 [running]:
testing.tRunner.func1.2({0x643360, 0xa3b3a0})
	/opt/go/src/testing/testing.go:1631 +0x24a
testing.tRunner.func1()
	/opt/go/src/testing/testing.go:1634 +0x377
panic({0x643360?, 0xa3b3a0?})
	/opt/go/src/runtime/panic.go:770 +0x132
github.com/ceph/go-ceph/cephfs/admin.TestCloneSubVolumeSnapshot(0xc000019d40)
	/go/src/github.com/ceph/go-ceph/cephfs/admin/clone_test.go:132 +0x80b
testing.tRunner(0xc000019d40, 0x6ad750)
	/opt/go/src/testing/testing.go:1689 +0xfb
created by testing.(*T).Run in goroutine 1
	/opt/go/src/testing/testing.go:1742 +0x390
FAIL	github.com/ceph/go-ceph/cephfs/admin	0.180s
FAIL
*** ERROR: returned 1
*** ERROR: cephfs/admin tests failed
make: *** [Makefile:221: test-containers-test] Error 1
Error: Process completed with exit code 2.

https://github.com/ceph/go-ceph/actions/runs/8494795424/job/23270374279
https://github.com/ceph/go-ceph/actions/runs/8502463560/job/23286663948
https://github.com/ceph/go-ceph/actions/runs/8516046284/job/23324505623

[phlogistonjohn]

Since these are all on main I suspect the recent issues loading the mgr modules.
A revert is in the works. See: ceph/ceph#56617

Let's retest main after the PR I linked has been merged.

[anoopcs9]

Looks like its all back green. Closing the issue.