Skill generator creates skills that breaks progressive disclosure patterns

[bopm]

Code of Conduct

• I agree to follow this project's Code of Conduct

AI Policy

• I agree to follow this project's AI Policy, or I agree that AI was not used while creating this issue.

Versions

does not matter, recent HEAD

Operating system

does not matter

Current Behavior

On skill generation, all information is being dumped into SKILL.md itself, which leads to the breaking of Progressive disclosure patterns and fast consumption of context by resulting skill.

Instead of that, the generated skill should contain multiple separate reference files for specific frameworks in the group, and follow other best practices from Skill authoring best practices.

Reproduction

Add multiple ash frameworks into the project, add usage_rules, and configure it for skill generation. Generate skill and look at wc -l for generated file. It'll be closer to 3k+ lines, completely clogging the context of any session.

Expected Behavior

Skills should stay under 500 lines, details should go to references, and references should contain a table of contents.

[bopm]

One of the ways to get an improved version of that generated skill is to install https://github.com/full-stack-biz/claude-skills-toolkit as a Claude plugin and ask it to refine the skill (/skill-refiner) to extract references. It can also be used as a temporary measure while the bug is present to keep any project context consumption under control.

[zachdaniel]

This is not a bug, it is something that is controlled by the library, not usage_rules itself. If the library author writes this:

usage-rules/
  foo.md
  bar.md

They will be built in as references and linked to from the main SKILL.md file.

[zachdaniel]

Oh, also you can include projects in references instead of in the main file via {<source>, :reference}

  defp usage_rules do
    # Example for those using claude.
    [
      file: "CLAUDE.md",
      # rules to include directly in CLAUDE.md
      usage_rules: ["usage_rules:all"],
      skills: [
        location: ".claude/skills",
        # build skills that combine multiple usage rules
        build: [
          "ash-framework": [
            # The description tells people how to use this skill.
            description: """
            Use this skill working with Ash Framework or any of its extensions.
            Always consult this when making any domain changes, features or fixes.
            """,
            # Include all Ash dependencies
            usage_rules: [:ash, {~r/^ash_/, :reference}] #<- like this
          ],
          "phoenix-framework": [
            description: """
            Use this skill working with Phoenix Framework.
            Consult this when working with the web layer, liveviews, controllers, views
            """,
            # Include all Phoenix dependencies
            usage_rules: [:phoenix, {~r/^phoenix_/, :reference}]
          ]
        ]
      ]
    ]
  end

[bopm]

Whoever writes the skill file is the producer of the skill. Skills should never be bigger than 500 lines. So I still find this one to be a bug. The default configuration produced a skill file for Ash for me with 3k lines. That is bad.

[zachdaniel]

Are you perhaps looking to open an issue in the Ash repo? usage_rules itself does not define the Ash rules. For Ash itself, the fix would be to create sub-rules in the pattern I described above.

[bopm]

My point is, you have control of the infliction point; fixing the issue here may save years of computing time instead of leaving it hanging in the default configuration while waiting for every single library author who utilizes this library's conventions. People lose money on this. Nature gets affected. You have the best instrument to solve it fast.

[bopm]

And let's not even start about how many library authors will overlook this if it'll be default flow. So it'll just keep happening over and over again. Ideally, I would call for some kind of hard linter for produced skills if this feature is to stay (which I hope it is).

[zachdaniel]

This library only works for libraries that define usage rules files.

More concretely, you referred to it as a "bug" but it sounds like you're requesting some kind of feature. Do you have suggestions for how we could solve the problem of library authors defining usage rules files that are too large? Our "solution" is supporting defining multiple sub-rules, which are cohesive units that can be linked in reference.md. That feature is released, and the plan is to implement that across the Ash projects. What else can we do?

[bopm]

Your library produces skills that burn down tokens because it's not what skills are supposed to be. I given you an official reference on that, where it explains why. I did notice that with my own eyes over the last few days, looking at an old ongoing project of mine. I showed you what should be done, instead of what you do, and I created a ticket in the codebase that produces problematic skills. It is not a feature. It's a bug.

[zachdaniel]

I'm sorry but you are clearly not reading at what I am saying. I am not arguing that skills should be over 500 characters. We are planning on breaking up the ash project usage rules files soon for exactly this reason. But you have opened an issue in the usage_rules repository.

---

It is not this libraries job to know about or fix the skills of other packages. The package author decides the shape of their usage rules. We transform it and place it in either skills or AGENTS.md. There is no magical way for usage_rules to know how you mean to split up your context.

If you have a problem with the Ash usage rules, then please open an issue on the Ash repository requesting that it be broken up into smaller files so that it is more conducive to generating skills.

This library does not "define" skills. It transforms text files from a library into text files in your project.

[bopm]

This library produces something that destroys the planet and causes financial losses for its users. If that behaviour is a feature, I have some sad news for you.

[zachdaniel]

Can you, in your own words, please describe to me how exactly this package functions?