Eliminate reflection for metadata provider discovery to improve startup time and native image support

[stalep]

Summary

The _AeshMetadata classes generated by the annotation processor are currently discovered via MetadataProviderRegistry using Class.forName() with a naming convention (e.g., MyCommand → MyCommand_AeshMetadata). This has two costs:

1. Reflection overhead — Class.forName() on every command class during registry build
2. Native image configuration — Every generated metadata class needs an explicit entry in reachability-metadata.json (jbang has 65 such entries)

Since startup time is critical for aesh (5.5x faster than picocli on native image), eliminating this reflection could further improve cold-start performance.

Current behavior

AeshCommandContainerBuilder.create(MyCommand.class)
  → MetadataProviderRegistry.lookup("MyCommand_AeshMetadata")
    → Class.forName("com.example.MyCommand_AeshMetadata")   // reflection
      → provider.newInstance()                                // reflection

Each Class.forName() call involves classloader delegation, security checks, and on native image requires pre-registered reflection metadata.

Alternative approaches

Option 1: Java ServiceLoader (recommended)

The annotation processor generates a META-INF/services/org.aesh.command.CommandMetadataProvider file listing all metadata classes:

com.example.MyCommand_AeshMetadata
com.example.OtherCommand_AeshMetadata

Discovery becomes:

ServiceLoader.load(CommandMetadataProvider.class)

Advantages:

• Standard Java mechanism, well-understood
• GraalVM has built-in ServiceLoader support — auto-discovers services at build time without manual reflection entries
• Eliminates all 65+ reflection entries from native-image config
• Works with incremental compilation (processor appends to services file)

Considerations:

• Need to handle the services file across incremental and multi-module builds
• ServiceLoader loads all providers eagerly; may need a keyed lookup (provider reports which command class it handles) to avoid loading unused providers
• The processor would need to use Filer.createResource() to generate the services file

Option 2: Generated static registry

The annotation processor generates a single registry class that directly instantiates all providers:

public class _AeshMetadataRegistry {
    private static final Map<Class<?>, CommandMetadataProvider> PROVIDERS = new HashMap<>();
    static {
        PROVIDERS.put(MyCommand.class, new MyCommand_AeshMetadata());
        PROVIDERS.put(OtherCommand.class, new OtherCommand_AeshMetadata());
    }
    public static CommandMetadataProvider get(Class<?> cmdClass) {
        return PROVIDERS.get(cmdClass);
    }
}

Advantages:

• Zero reflection — direct new calls
• Single class to register in native-image config (if any)
• Fastest possible lookup (HashMap get)
• No ServiceLoader overhead

Disadvantages:

• Incremental compilation is harder — the registry must be regenerated when any command changes
• Multi-module builds need coordination (each module has its own registry)
• Tight coupling between the registry and all command classes

Option 3: Direct instantiation in generated code

Instead of looking up the metadata provider externally, each command class could have a static method or field:

// Generated by processor alongside MyCommand_AeshMetadata
public class MyCommand implements Command<CommandInvocation> {
    // ...existing code...

    // Added by processor via source generation or bytecode weaving
    public static CommandMetadataProvider _aeshMetadata() {
        return new MyCommand_AeshMetadata();
    }
}

Then AeshCommandContainerBuilder calls MyCommand._aeshMetadata() directly.

Advantages:

• Zero reflection, zero ServiceLoader
• Natural per-class locality
• Works with incremental compilation

Disadvantages:

• Cannot add methods to user classes via annotation processing (would need a different approach like a companion class or interface)
• Could use a convention: MetadataProviderRegistry first checks if the class implements a HasMetadataProvider interface before falling back to Class.forName()

Option 4: Hybrid — direct instantiation with reflection fallback

Keep the current Class.forName() approach as a fallback, but add a fast path:

public static CommandMetadataProvider lookup(Class<?> cmdClass) {
    // Fast path: check if the metadata class is already on the classpath
    String metadataClassName = cmdClass.getName() + "_AeshMetadata";
    try {
        // Use the classloader that loaded the command class
        Class<?> metadataClass = cmdClass.getClassLoader().loadClass(metadataClassName);
        return (CommandMetadataProvider) metadataClass.getConstructor().newInstance();
    } catch (ClassNotFoundException e) {
        // Fallback to reflection-based builder
        return null;
    }
}

This is essentially the current approach but makes the reflection more explicit. Not a real improvement.

Recommendation

Option 1 (ServiceLoader) is the best balance of simplicity, standards compliance, and native image support. It eliminates all reflection entries with minimal code change.

If startup time is the absolute priority and ServiceLoader's eager loading is a concern, Option 2 (static registry) is the fastest but harder to maintain across incremental builds.

Impact estimate

For jbang with 65 metadata classes:

• Eliminates 65 Class.forName() calls during startup
• Removes 65 lines from reachability-metadata.json
• On native image, removes reflection initialization overhead for 65 classes

Context

Found during jbang picocli-to-aesh migration (jbangdev/jbang#2453). The _AeshMetadata reflection entries are the largest single category in jbang's native-image configuration.

[stalep]

Analysis & Plan

Current State

Option 1 (ServiceLoader) is already implemented:

• MetadataProviderRegistry uses ServiceLoader.load(CommandMetadataProvider.class) for discovery (no Class.forName() in registry or AeshCommandContainerBuilder)
• AeshAnnotationProcessor.writeServiceFile() generates META-INF/services/org.aesh.command.metadata.CommandMetadataProvider at compile time
• AeshCommandContainerBuilder.create() checks MetadataProviderRegistry.getProvider() first, falls back to reflection only if no provider exists
• Zero Class.forName() calls in the metadata discovery path

Remaining Issues

Despite ServiceLoader being in place, three issues remain:

Issue 1: Eager Loading of All Providers (Medium-High)

MetadataProviderRegistry.loadProviders() iterates all ServiceLoader entries on first access:

for (CommandMetadataProvider provider : loader) {
    map.put(provider.commandType(), provider);
}

For jbang (65 commands), this means:

• 65 metadata classes loaded (each with a static initializer that calls CLConverterManager.getInstance(), getDeclaredField(), setAccessible(true), creates converter/completer instances)
• 65 command classes loaded (because provider.commandType() returns CommandClass.class, forcing class loading)
• All of this happens even if the user runs a single command

This partially negates the startup-time benefit of the generated path.

Issue 2: Incremental Compilation (High for Gradle users)

writeServiceFile() overwrites the META-INF/services file with only the providers generated in the current compilation round. Under incremental javac, modifying one command class produces a service file listing only that one provider, losing all others.

The processor declares no Gradle incremental annotation processing support, so Gradle forces full recompilation on every change -- safe but slow.

Issue 3: No GraalVM Native Image Configuration (Medium)

No reflect-config.json, resource-config.json, or META-INF/native-image/ exists in the project. While GraalVM auto-detects ServiceLoader.load() with constant class literals, the generated metadata providers call getDeclaredField() / setAccessible(true) in their static initializers (for private fields), which requires explicit reflection entries in native-image config.

---

Evaluation of Options Against Remaining Issues

Option 1: ServiceLoader -- Already Done

Status: Implemented. The remaining issues above are orthogonal improvements to the existing ServiceLoader implementation.

Option 2: Generated Static Registry -- Not Recommended

A single _AeshMetadataRegistry class with a static {} block that instantiates all providers would:

• Still eagerly load all providers (same problem as Issue 1, just without ServiceLoader overhead)
• Be worse for incremental compilation -- regenerating the registry requires all command classes be available in the compilation round
• Be worse for multi-module -- each module gets its own registry, requiring a merge strategy at runtime
• Only save ServiceLoader overhead (~microseconds), not the real cost (class loading + static initializers)

Verdict: Solves nothing the current ServiceLoader approach doesn't, adds complexity. Skip.

Option 3: Direct Instantiation (HasMetadataProvider interface) -- Not Feasible

Having commands implement an interface with a static metadata method:

• Cannot be added by annotation processors (processors generate separate files, not modify user classes)
• Would require bytecode weaving (e.g., ASM) or a Gradle/Maven plugin -- heavy infrastructure
• Even with a companion check, still needs the ServiceLoader fallback

Verdict: Not feasible with annotation processing alone. Skip.

Option 4: Hybrid -- Already the Current Design

The current AeshCommandContainerBuilder.create() already does exactly this:

1. Check MetadataProviderRegistry.getProvider() (ServiceLoader-based, no reflection)
2. Fall back to doGenerateCommandLineParser() (runtime reflection) if no provider

Verdict: Already implemented. Nothing to do.

---

Recommended Plan: Fix the Three Remaining Issues

Phase 1: Lazy Provider Loading (fixes Issue 1)

Replace eager ServiceLoader iteration with lazy, on-demand loading using ServiceLoader.stream() (Java 9+).

Approach -- naming convention index:

The processor already generates MyCommand_AeshMetadata from MyCommand. We can reverse this using ServiceLoader.stream() + Provider.type():

private static Map<String, ServiceLoader.Provider<CommandMetadataProvider>> loadLazy() {
    Map<String, ServiceLoader.Provider<CommandMetadataProvider>> map = new HashMap<>();
    ServiceLoader<CommandMetadataProvider> loader = ServiceLoader.load(CommandMetadataProvider.class);
    for (ServiceLoader.Provider<CommandMetadataProvider> sp : loader.stream().toList()) {
        // Provider type: "com.example.MyCommand_AeshMetadata"
        // Command class: "com.example.MyCommand" (strip suffix)
        String providerName = sp.type().getName();
        String cmdName = providerName.substring(0, providerName.length() - "_AeshMetadata".length());
        map.put(cmdName, sp);
    }
    return map;
}

Key details:

• sp.type() returns the provider Class object without running its static initializer (just class metadata loading)
• The _AeshMetadata suffix is stripped to derive the command class name
• Only when getProvider(SomeCommand.class) is called does sp.get() instantiate the provider (running static initializers, loading converters, etc.)
• For inner class commands like Config$Set, the metadata class is Config_Set_AeshMetadata -- the reverse mapping needs to handle $ vs _ for nested classes

To handle the inner-class caveat cleanly, also add a commandTypeName() method to the interface:

// In CommandMetadataProvider:
String commandTypeName(); // returns e.g. "com.example.Config$Set"

The processor generates it as a compile-time string literal (no class loading needed). But since calling it requires instantiation, it serves as a validation fallback after sp.get(), not as the primary index key.

Impact: For jbang, reduces startup from loading 65 command + 65 metadata classes to ~5 (the command actually invoked + its parent group + mixins).

Phase 2: Incremental Compilation Support (fixes Issue 2)

1. Read-and-merge service file: Before writing META-INF/services, attempt to read the existing file via filer.getResource() and merge new entries with existing ones (same pattern used by Google's AutoService).

2. Declare Gradle aggregating processor support: Create META-INF/gradle/incremental.annotation.processors with:

   org.aesh.processor.AeshAnnotationProcessor,aggregating
   

   This tells Gradle the processor can handle incremental builds.

Impact: Faster incremental builds for Gradle users. Correct service file for incremental javac.

Phase 3: GraalVM Native Image Support (fixes Issue 3)

Generate native-image configuration during annotation processing:

1. resource-config.json -- ensures the service file is included in the native image
2. reflect-config.json -- entries for private fields accessed via getDeclaredField() in generated providers. The processor already knows which fields are private (it generates the reflection calls), so it can emit the config alongside the code.

Place these under META-INF/native-image/org.aesh/<artifactId>/ in each module's output.

Alternative (longer term): Eliminate getDeclaredField() entirely from generated code by using MethodHandle/VarHandle with Lookup.privateLookupIn() -- but this is a larger change.

Impact: Zero manual native-image configuration needed for projects using aesh-processor. Eliminates the 65 reachability-metadata.json entries jbang currently needs.

---

Priority Order

Phase	Issue	Effort	Impact	Priority
1	Lazy provider loading	Medium	High (startup time)	P1
2	Incremental compilation	Low-Medium	Medium (build time)	P2
3	Native image config	Medium	High (DX for native)	P2

Phase 1 is the most impactful for the stated goal (startup time). Phases 2 and 3 are important for developer experience but don't affect runtime performance.

Files to Modify

Phase 1:

• aesh/src/main/java/org/aesh/command/metadata/CommandMetadataProvider.java -- add commandTypeName()
• aesh/src/main/java/org/aesh/command/metadata/MetadataProviderRegistry.java -- lazy loading with ServiceLoader.stream()
• aesh-processor/src/main/java/org/aesh/processor/CodeGenerator.java -- generate commandTypeName() override

Phase 2:

• aesh-processor/src/main/java/org/aesh/processor/AeshAnnotationProcessor.java -- read-and-merge service file, Gradle incremental support

Phase 3:

• aesh-processor/src/main/java/org/aesh/processor/AeshAnnotationProcessor.java -- generate native-image config files
• aesh-processor/src/main/java/org/aesh/processor/CodeGenerator.java -- collect private field metadata for reflect-config