Align public namespace with scikit-learn for scikit-learn elements

[csadorf]

Ensure that our import paths for scikit-learn elements largely match those of scikit-learn. For example, the accuracy_score function should be documented as importable from cuml.metrics, but not from the internal cuml.metrics.accuracy module.

• Elements are importable from the equivalent public namespace.
• Elements are not importable from other public namespaces.

See related discussion that motivated this issue.

What is considered a public namespace is defined here.

[betatim]

I came back to this triggered by the LabelEncoder and TargetEncoder renaming in #7798.

This issue is about clean up like the LabelEncoder one or the accuracy_metric mentioned in the top comment. I think as part of it we should consider the idea of making it more obvious what is and isn't part of the public API.

For goal one we should create a test that enforces this so that we don't regress. Otherwise I think it is straightforward to achieve this goal.

The second goal "should not be importable from other places" is a bit tricky to achieve because Python is so permissive. The best we can do is to mark things as private. For example moving cuml.metrics.accuracy to be cuml.metrics._accuracy. How do others see this?

The most controversial aspect is the "shortcuts" that exist in cuml right now. For example cuml.RandomForestClassifier exists as a shortcut to cuml.ensemble.RandomForestClassifier. Some people like having these shortcuts because it removes the need to look up which exact sub-module something is located in. Others think that this causes unneeded noise in the namespace. I don't know how to resolve this (it feels like something that could be a matter of taste/opinion, not an engineering question).

I'm in favour of removing the shortcuts. My argument is that the overhead of remembering where something is is small and that code should be optimised for reading, not writing. For reading it helps to have the context that namespaces give you ("Namespaces are one honking great idea -- let's do more of those!"), it makes the main namespace cleaner (helping tab complete), it makes it simpler to have tools that inspect our namespace if things are not available in several places.

How do we resolve this? A vote? Coin flip? Ask AI to decide?

---

A short diff illustrating how you can make the fictitious cuml.foo module private https://github.com/rapidsai/cuml/compare/main...betatim:cuml:namespace-cleanup?expand=1 It continues to make cuml.foo available, based on cuml._foo, but raises a warning. This gives us a way to make a module private over one or two releases. This is a useful tool to have for modules that are not obviously private already, like cuml.internals. For these we don't need a deprecation cycle, you should not have been using them.

I've not tried it yet but am 99.9% sure we could have a similar pattern for cuml.RandomForestClassifier - it exists but raises a warning when accessed. Then after a certain number of releases with warning we remove it.

[csadorf]

I think there are two things we need to address here:

1. Define the public API. All private modules must be prefixed with an underscore. All other modules are public by definition and must define __all__. This way there is no ambiguity — the public API can be determined by following the __all__ declarations starting from the root package. This applies equally to APIs with an upstream counterpart and to cuML-only APIs. Namespaces named experimental are public, but are considered unstable and not subject to the deprecation policy.

2. Align import paths with the upstream libraries. All classes and functions that have a clear counterpart in the upstream library (scikit-learn, umap-learn, hdbscan) should be importable from the exact same path. For example, cuml.ensemble.RandomForestClassifier should work just as sklearn.ensemble.RandomForestClassifier does.

I'm in favour of removing the top-level shortcuts (e.g., cuml.RandomForestClassifier) for all the reasons Tim already mentioned. They add noise to the root namespace and make it harder for tooling to inspect the public API. The deprecation-shim approach in @betatim's branch generally looks like a good mechanism for phasing these out (although we should not redirect users to internal namespaces without a strong warning to not do that). All names and namespaces that were reasonably considered public get a FutureWarning for one release per our deprecation policy, then are removed.

To prevent regressions, we should add static checks. Ruff already covers some of the basics out of the box:

• F822 flags names in __all__ that aren't defined in the module (catches stale entries after renames).
• PLE0605 ensures __all__ is a valid list/tuple of strings.
• F401 flags unused imports unless they appear in __all__ (enforces that re-exports are declared).

For some others we'd need a few custom checks (some could be removed once ruff supports them):

• All public packages define __all__ (foo.bar without __all__ would be flagged, but foo._bar or foo._bar.baz would not).
• All names listed in __all__ are documented in the user-facing API reference.
• All submodules not listed in a higher-level __all__ are prefixed with an underscore.

I don't think we have to rename all internal names as long as we consistently document the public API via __all__.

We should also update our Python developer documentation to be clearly aligned with this.

I would recommend the following implementation approach:

1. Update developer docs, consistently define __all__ to define the public API, and add all related checks. If there is ambiguity we can sort it out with minimal user impact.
2. Rename all private modules with deprecation where necessary. This should be coordinated with other infra-PRs since there would be a ton of merge conflicts.

I am against a phased or partial roll-out where we bring some namespaces into compliance and others not. The whole point of this effort is to increase clarity and consistency and reduce ambiguity. A partial roll-out would only add to the confusion.

[csadorf]

The discussion on public vs private namespaces should probably be continued in #7482 .

[betatim]

About custom ruff rules: it seems like you can't make plugins/custom rules for ruff astral-sh/ruff#8409 but we could maybe use a pre-commit hook or add a unittest to achieve the same thing. I agree that without such an automated check we'd regress quickly.