Check code examples in documentation

[MartinSStewart]

What the rule should do:
Find code examples in function and module documentation and verify that it's syntactically correct (with a few exceptions) and that the functions and types that are used exist.

This is similar to https://github.com/stoeffel/elm-verify-examples but where that checks that the code computes a specific value, this rule focuses on if the example will (probably) compile

What problems does it solve:
I've spent a lot of time double checking that the code examples in packages I've written actually work. Usually I end up missing something after I've made API changes making the examples a hindrance to newcomers.

Example of things the rule would report:

Example 1:

{-| My function. 
    
    numberTwicer 4 -- 8

-}
numberDoubler a = a * 2

Here we changed the function name so numberTwicer no longer exists. We know that this is a line of code and not a comment because it's indented 4 spaces.

Example 2:

{-| My function. 
    
    numberTwicer MyFunction.four -- 8

-}
numberDoubler a = a * 2

Here we are using MyFunction.four without importing it. This rule would offer this fix:

{-| My function. 

    import MyFunction    

    numberTwicer MyFunction.four -- 8

-}
numberDoubler a = a * 2

Example of things the rule would not report:

Example 1:

{-| This isn't a code example so it shouldn't get checked. -}
foo = 0

Example 2:

{-| 

    numberDoubler 0 -- 0
    numberDoubler 1 -- 2
    numberDoubler 2 -- 4

 -}
numberDoubler a = a * 2

Multiple expressions separated by newlines is valid.

Example 2:

{-| 

    import MyModule

    numberDoubler 0 -- 0
    numberDoubler 1 -- 2
    numberDoubler 2 -- 4

 -}
numberDoubler a = a * 2

Imports followed by multiple expressions separated by newlines is valid.

Example 3:

{-| ...

## Failure

    port module Main exposing (main)

    import Html
    import Json.Encode as Encode


    -- Port `action` is never used.
    port action : (Encode.Value -> msg) -> Sub msg

    port alarm : Encode.Value -> msg

    -- Port `alarm` is never used, because `play` is never used.
    play : Cmd msg
    play =
        alarm (Encode.string "play")

    main =
        Html.text "Hello"

-}

Normal module code is also allowed.

When (not) to enable this rule:

This rule is probably going to end up being pretty complex to handle different use cases. There's probably some that will be too difficult to support or conflict with other use cases. For those situations you wouldn't want to use this rule.

I am looking for:

If people could provide code examples where it's not clear if the rule would consider them invalid or not. There are probably a lot of edge cases something like this would need to handle.
Also any thoughts in general.

[sparksp]

This is interesting because you're talking about standardising documentation examples and essentially creating a new sub-language of Elm. This means that it may not be possible to use something like elm-syntax to help verify the code because we won't know whether the example is a whole file, a declaration or an expression - or we may have to try and fail until something works. Some questions from me...

Question 1 - If multiple expressions can be separated by new lines then can you give a multi-line example? These usually come up for me when I want to add context to an example.

{-| Ensure consistent use of import aliases throughout your project.

    config : List Rule
    config =
        [ NoInconsistentAliases.config
            [ ( "Html.Attributes", "Attr" )
            ]
            |> NoInconsistentAliases.detectAliases
            |> NoInconsistentAliases.rule
        ]

-}

Question 2 - When would imports be considered missing? In this example from NoAlways.elm should I import NoAlways? Should I also import something exposing Rule?

{-| Forbid the use of [`always`](https://package.elm-lang.org/packages/elm/core/latest/Basics#always).

    config : List Rule
    config =
        [ NoAlways.rule
        ]
-}

Question 3 - How would we give examples of a complete Elm file? This example should be valid code (although an example of a rule violation).

{-| ...

## Failure

    port module Main exposing (main)

    import Html
    import Json.Encode as Encode


    -- Port `action` is never used.
    port action : (Encode.Value -> msg) -> Sub msg

    port alarm : Encode.Value -> msg

    -- Port `alarm` is never used, because `play` is never used.
    play : Cmd msg
    play =
        alarm (Encode.string "play")

    main =
        Html.text "Hello"

-}

I realise all of my examples are from rule documentation themselves, but that's all I have published and is where I usually give my best examples. I don't often give examples for internal modules.

[MartinSStewart]

This is interesting because you're talking about standardising documentation examples and essentially creating a new sub-language of Elm.

To some extent I think this idea has already been explorered a bit. elm-format will format code inside of documentation and it seems to be more lenient with it than it is with normal Elm code.

Question 1 - If multiple expressions can be separated by new lines then can you give a multi-line example? These usually come up for me when I want to add context to an example.

Good point, I think it makes the most sense if new expressions must start with the same indentation as the first expression. Everything else is treated as part of a multiline expression.

{-|
    foo 0 -- Single line expression
    foo 1 
        |> bar -- Part of the previous expression because it's indented in more.
-}

Also, your example here isn't an expression but instead a function declaration. Having multiple function declarations is already valid in elm-syntax.

Question 2 - When would imports be considered missing? In this example from NoAlways.elm should I import NoAlways? Should I also import something exposing Rule?

When encountering a function call or type, the rule would first check if it's defined anywhere in the module containing this documentation. If it doesn't find it, then it checks for imports to see if it's exposed there. Lastly it checks if the exposed function/type is defined in that imported Module.

In your example, you'd need to change it to

{-|
    import Review.Rule exposing (Rule)
    import NoAlways

    config : List Rule
    config =
        [ NoAlways.rule
        ]
-}

Question 3 - How would we give examples of a complete Elm file? This example should be valid code (although an example of a rule violation).

I think I should have provided more examples but normal function declaration and entire modules is also allowed. I just focused on expressions because normally expressions have to be inside a function declaration.

[sparksp]

Question 1 - If multiple expressions can be separated by new lines then can you give a multi-line example? These usually come up for me when I want to add context to an example.

Also, your example here isn't an expression but instead a function declaration. Having multiple function declarations is already valid in elm-syntax.

I think I should have provided more examples but normal function declaration and entire modules is also allowed. I just focused on expressions because normally expressions have to be inside a function declaration.

I think that's what got me a bit stuck, because everything in the opening issue talked about expressions; that's why I gave some examples of declarations and files. When I have had multiple expressions I've tended to separate them by a blank line.

I presume this would also allow the elm-verify-examples style of attached output -->, since these are just comments?

elm-format will format code inside of documentation and it seems to be more lenient with it than it is with normal Elm code.

I think elm-format is designed to handle incomplete code and try to do its best with it, that probably helps it quite a bit in terms of code samples. However we'd be looking for "correctness" too.

[MartinSStewart]

I think that's what got me a bit stuck, because everything in the opening issue talked about expressions

I've added your module example. Hopefully that makes this more clear.

I presume this would also allow the elm-verify-examples style of attached output -->, since these are just comments?

Yeah. This rule wouldn't verify those but elm-verify-examples could work along side this rule to handle that.