ksconf for splunk cloud migration. Need pointers.

[bayeslearner]

The problem

We are about to move the entire Splunk infrastructure to Splunk Cloud. Splunk Cloud does not allow deploying local folders. ksconf seems to be a perfect tool to facilitate the process of migration and future maintenance.

However, the documentation on this topic seems still sparse. Greatly appreciate any hint on the following questions.

Details

Does the directory have to be organized in the structure described in the documentation? As a hypothetical simple example. say 1 sh (sh1), and 1 indexer (indx1) are nodes from the existing clusters that all have apps/KOs that use both default/local folders, and they also have overlapping as the following:

sh1/
-- Splunk-TA-app01 <-- splunk/splunkbase apps, (git-versioned?)
----default
------props.conf
----lookups
-- z_xxx_TA_app01 <--- this is the customization of the Splunk-TA-app01 (git-versioned)
----default
------xxx.conf
----local <-- user added stuff in here from GUI, probably not a good idea
------xxx.conf
----lookups
-- zpp02 <-- user app,
----default
------yyy.conf
----local
------yyy.conf
----lookups

idx1/
-- z_xxx_TA_app01
----default
------xxx.conf
----local
------xxx.conf
----lookups
-- zpp03
----default
------props.conf
----local
------props.conf
----lookups

Could the target be something like this:
cloud_deploy/
-- Splunk-TA-app01 (versioning)
----default
------props.conf
----lookups
-- z_xxx_TA_app01 (versioning)
----default
------xxx.conf
----lookups
-- zpp02 (versioning)
----default
------props.conf
----lookups
-- app03 (versioning)
----default
------props.conf
----lookups

cloud_dist/
-- app01.spl
-- app02.spl
-- app03.spl
-- app04.spl

1. Is the above structure reasonable for ksconf to do its work or do I have to convert them into a layering structure first?

2. Can you provide some examples on how to minimize, clean/merge based on this hypothetical case?

3. Is this structure sufficient for ongoing cloud maintenance?

Thank you again!

[lowell80]

1. You should never be forced into using layers. There may be times where it's the most practical solution, but it isn't required.
   2 + 3. I'd probably need more detail before I could go deeper on this. I have 2 general comments. (1) we offer consulting services where we could go much deeper, look at tangible examples and not be limited to public conversation. (2) With Splunk Cloud there's an ongoing challenge with local changes. (This is technically true on any Splunk Enterprise deployment too, but there you can get around that problem with file system access to the server artifacts.). The alternative is to make sure that changes are never made to these app from within Splunk (via the UI, or REST) and therefore all changes go through a version control (and likely CI/CD) pipeline for all changes. Pulling changes back out of Splunk Cloud is challenging because without file-system access, all you have is the REST endpoints, and I've developed techniques for pulling data out of there but (a) it cannot ever be fully clean based on the merged nature of how .conf files are handled, and (b) there's no way to tell the difference between what's in local and default. (Well, unless you fully control the app yourself, then you can find the difference with say "ksconf minimize" but, that doesn't really solve problem [a] fully.)

Sorry for the delayed and rambling response. I hope it helps.