Programmatically generate ellie links for showcase examples

[supermacro]

Problem: When creating new code examples in the showcase app, we need to specify a metaInfo record with a ellieDemo url (see code snippet below for an example). The issue is that for a lot of these demos, the code is not yet published in https://package.elm-lang.org/ ... so we have to then do a follow up PR / commit after publishing to https://package.elm-lang.org/ to update the ellieDemo link to actually contain the demo code.

Right now I just enter a dummy url and forget to update it (plus, it's extra work and I'm lazy to follow up and update it).

basicExample : Model -> Styled.Html Msg
basicExample model =
    let
        metaInfo =
            { title = "Basic"
            , content = "Basic usage of checkbox."
            , ellieDemo = "https://ellie-app.com/9mjDjrRz2dBa1"
            }
    in
    Container.createDemoBox
        (DemoBoxMsg << SimpleExample)
        model.basicExample
        (\_ -> SimpleExample.example)
        metaInfo

Desired Outcome

1. Use Ellie's programmatic API to dynamically create ellie demos programmatically.

Documentation on how to do this.

Specifically, use Luke Westby's suggestion of encoding the entire elm demo app in the URL.

2. Update all existing demos to use this new API

[joshuanianji]

Ooh this looks really interesting. I'd like to take a shot at this!

I was just wondering how you would like to approach this. Would this be python file (or whatever executable) we'd run to generate an ellie link given the file, and print out the link? Or maybe it doesn't matter as long as the link generation is slightly automated haha.

On a side note, do you want to add a hacktoberfest topic to this repo? It might attract some more contributors and people who can help :)). Also my PRs don't count toward Hacktoberfest right now because this repo isn't opted in 😅.

[supermacro]

Go for it!

I was just wondering how you would like to approach this.

We would generate the url in elm once the source code is loaded from the file-server. So clicking the code ellie icon in a demo box prior to the source code being loaded should be a no-op. Once we generate the correct url as per the issue comment I linked above, then we would add that as the ellieDemo value for metaInfo.

Does that make sense?

---

do you want to add a hacktoberfest topic to this repo

Done!

[joshuanianji]

We would generate the url in elm

Oh I think I get it! What I understand is that the Ellie link will be created in Elm by the file contents, once it's received by the ExampleSourceCodeLoaded. That makes it a lot simpler, actually.

Done!

Thank you so much!

[joshuanianji]

I'll be busy over the weekend but I just wanted to give you an update on what I did recently (Sorry for the long comment!)

TL;DR: URL encoding is finished (with gist and github repo) but there are some caveats that prevent us from using it right away.

Good news

URL encoding is surprisingly easy, since all you have to do is replace some reserved characters with their percent encodings. I made this gist that showcases that functionality. It hardcodes the Ellie title, HTML and the packages that the Ellie link will generate, but those aren't too difficult to change.

For example, a simple Hello World example:

module Main exposing (main)

import Html exposing (Html)


main : Html msg
main =
    Html.text "Hello World!"

Will be converted to this Ellie link.

If you want to test it out yourself in the command line, I made a small headless Elm program that prints out the URL when the file is taken as a CLI argument. The repo is here.

Caveats

This seems close to actual functionality, but it unfortunately doesn't work on the antd code examples, without some major refactoring. The program assumes the code is already compatible with Ellie straight off the bat, so file names that aren't called Main and aren't exposing a main function wouldn't work.

I looked into the examples and it seems like there are three main types of "files":

1. Stateless and Message-less (most common)

module Routes.ButtonComponent.TypeExample exposing (example)

Files like these are pretty easy to make compatible because we just have to rename the file to Main. Renaming example to main is harder but we could just add a line below the imports like so (although this is a little hacky):

main : Html msg 
main = example

2. Stateless with Messages

module Routes.ButtonComponent.IconExample exposing (Msg, example)

This will be harder to make compatible because, along with changing the name and exports, we'd have to add some extra code to make this able to export a main : Program () Model Msg function.

3. Stateful with Messages

module Routes.AlertComponent.CloseableExample exposing
    ( Model
    , Msg
    , init
    , update
    , view
    )

This is similar to the above problem, but we have lot more things already defined for our main function.

I believe these problems are all solvable with elm/parser, though it will take some time. I'm not sure how many changes you'd like to make to the example files, since the complexity of this problem might change slightly with how flexible you are willing to be. An example is changing the example function to main in all the files.

Apologies for the really long winded comment! This problem's really interesting to look into. I'll be busy with homework for this weekend so I won't be able to do much, but I hope this comment helps you out with this issue.

[supermacro]

Amazing! Don't apologize for the long comment. This is great research & detail!

Regarding the list of packages, I thought that it was optional to specify package versions with this hidden ellie API? We need to use the latest version ofelm-antd. The showcase is always running on the latest version of elm-antd. If we hardcode a elm-antd version then a lot of the new examples won't work unless we modify the version referenced within the file enocder / url generator.

So ideally there's an option to say Package "supermacro/elm-antd" "latest" or omit the package version entirely (which would then resolve to using the latest version).

One drawback with this approach is that ellie link generation won't work for local development if the example is using an unpublished API. But quite frankly I don't see why that matters.

---

Regarding the size & scope of this change:

I would recommend first implementing a naive approach / v1 implementation that only generates links for the "Stateless and Message-less" examples.

That way the size of the refactor is reduced by a considerable amount, and no one is biting off more than they can chew :)

Once that's merged, then we can iterate on this design and generalize it for the more complex examples.

An example is changing example to main in all the files.

I would say, do what you think makes sense in the context of a v1 implementation. We can then do some manual QA'ing to make sure things are still good.

[joshuanianji]

You're right, I feel like we should be able to specify the latest package version automatically, but unfortunately I couldn't find a way to do that with the URLs :((.

I looked at the source code for the Ellie URL parsers a bit more, specifically at Version.elm and Package.elm, and it seems that they only accept hardcoded numbers. Not specifying a package version at all would make the Package parsing fail completely.

Hardcoding the latest elm-antd version is going to be kind of annoying, but it seems to be the best we can do unless one of the Ellie devs has something to say about it.

---

I would recommend first implementing a naive approach / v1 implementation

A naive v1 implementation is definitely possible! I will be a little slow to make it due to a couple of midterms that are coming up, but I'll submit a PR with basic functionality sometime within a week and we can slowly work on making it more powerful.

[supermacro]

Hmm, ok I think another thing that could be done is to dynamically get the version of elm-antd and then use that to create the ellie links.

It might be possible to do this in elm using the metadata-utils package (but I am not very familiar with the package, so I can't make any promises).

Alternatively, the version could be hardcoded* at build time by fetching the version of elm-antd using github's API (and reading the list of tags). Then there would be a javascript environment variable that would use that value, and it would get sent into elm via ports.

---

*hardcoded is a bit of misnomer (but webpack basically does hardcode environment variables for anything that has the process.env.<ENV_VAR> pattern)

[joshuanianji]

I took a quick look at the options and it seems like Github's API is the easiest to use. I'm not too familiar with JS environment variables, though, especially with Webpack. Maybe you can make some changes once I have a PR ready 😅 .

Right now, I can just do an HTTP request (either in JS or on the Router.elm init function), and listen to the response on Router.elm. I'm not sure how I would "send" the version to a component without introducing a bunch of new code, though. I might have to introduce a "saveVersionToModel" field in the DocumentationRoute type. What do you think?

[supermacro]

I'm not too familiar with JS environment variables, though, especially with Webpack

No worries! It's quite straight forward. Let me see if I can explain in a simple manner:

JS that runs in browsers technically does not have environment variables. The process object is undefined.

However, in NodeJS, process is indeed defined, and you can access environment variables in NodeJS using process.env.<ENV_VAR_NAME>.

So what webpack does (remember, webpack runs in a NodeJS context), with the help of the awkwardly named "DefinePlugin" plugin, is that it allows you to replace tokens in your JS code with anything you want. So what I've set up is a string replace to take anything with the process.env.<ENV_VAR> pattern and replace it with the value of that environment variable at runtime.

So for example, when I write process.env.NODE_ENV in the JS source code, webpack turns it into the string literal "development" or "production".

See here for the actual line of code where the above happens.

Does that make sense?

---

With regard to HTTP requests, they should be done at build time, meaning in a Circle CI script. So your PR would do an HTTP request inside of the config.yml file that runs on deployments and PRs.

Side note: I can install the github cli in the Docker file so that the github cli is usable within CI/CD runs in Circle.

So imagine there was something like this:

# inside of config.yml

# create an env variable based on an api call to list git tags from most recent to oldest, and then pipe it to 'head' to attain only the most recent git tag
export CURRENT_ANTD_VERSION=$(github-cli list-tags supermacro/elm-antd | head -n 1)

Note that the above is a hypothetical usage of the github cli as I actually am not familiar with the cli's API.

... Then in JS, you would call process.env.CURRENT_ANTD_VERSION .. and tada! You now have the antd version avaialble to you!

[joshuanianji]

Wow, thanks for such a good explanation of the environment variables! I definitely understand env variables a lot more, and your string replace webpack plugin is really cool.

My experience with webpack (and I guess environment variables as well) only went as far as using create-elm app and firebase, so I'm glad I'm getting to learn so much from this project.

---

With regard to HTTP requests, they should be done at build time

Oh wow I really misinterpreted your earlier comments, sorry! I haven't had a lot of experience with CI/CD, but it seems like a really neat thing to learn, and has a lot of good documentation online.

For the Github CLI, It turns out that you don't even need to install anything, since the API is a simple fetch command from a URL which returns a JSON string. From what I've seen, you can run terminal commands right in the config.yml file. If I'm correct, this script should give return the newest version, without any dependencies:

curl -s 'https://api.github.com/repos/supermacro/elm-antd/releases/latest' |  \
    python3 -c "import sys, json; print(json.load(sys.stdin)['tag_name'])"

This works on my terminal, though the python script is a bit weird. Alternatively, the jq library can make it a lot simpler, though this requires a 3rd party library.

curl -s 'https://api.github.com/repos/supermacro/elm-antd/releases/latest' | jq -r '.tag_name'

Then in JS, you would call process.env.CURRENT_ANTD_VERSION

I'm looking at the Circle CI documentation and there seems to be a lot of different ways to export an environment variable. If it doesn't matter how we do it, I'll just wrap the terminal command with $(), if that's a correct thing to do.

---

My last question in this unfortunately long comment is about passing down the version number from the Showcase.elm to the containers, which are responsible for setting the Ellie links.
I think the easiest way would be for the Showcase.elm to receive it as a flag and send it to Router.elm. We will add an extra argument to a Component's view function, which will send it to a container through a modified DemoBoxMetaInfo record. How does that sound? I can probably get a PR ready within the next day or so for you to see the code.

[supermacro]

Happy to help :)

I just checked to see if either jq or python3 were installed in the docker image that I use, and it turns out that neither are currently available.

So I've taken the liberty of adding jq to the Circle CI environment.

In config.yml file you can now run the curl + jq command you posted.

The best way to test out if your code works is to run it inside of a docker container.

# download the latest docker image from dockerhub
> docker pull giorgio14/elm-antd-ci:latest

# "go into" the docker container in a bash session
> docker run -it giorgio14/elm-antd-ci:latest bash

After you run the above command, you'll be running an interactive bash shell within the docker container. Here you can prototype ideas, and once you're happy with them, then add them to the config.yml file.

---

I did however just realize that the actual building / deployment of the app is done outside of circle 😅 It's done via netlify, so I'm going to have to read a bit about programmatic builds for netlify. Meaning that right now this task is a bit blocked until I figure out how to build the app within circle and then upload it to netlify.

[supermacro]

If you want to, you can set up a free netlify account and learn how you can use the netlify cli to build a site and upload it to netlify (but not publish it ... I manually deploy builds on netlify).

[supermacro]

Oh, I forgot to answer your last question!

Yup, I think passing the version as a flag makes total sense. Happy to review a PR around this.

[joshuanianji]

Thanks for installing jq into the environment! The API call works in the bash session.

And sorry for not being active lately, school's been pretty annoying. I'll look at netlify when I have more time later, but right now I'll just do a quick PR showcasing what I've got for the URL generator.

---

I decided to go the parsing route because generating an AST gives the most flexibility with the code manipulation. It's pretty bare bones and the parsing code is a bit wacky, but it works on Stateless and Message-less files, such as the Alerts and Typography.

To pass the versions down, I added a "version" field to every component's Model. The initialModel of the DocumentationRoute then became a function that took in a version and returned a model, so the Router.init function immediately passed on the version number to every component model on initialization. I also hardcoded the version number in the flags, but this is pretty easy to replace with the environment variable once we get that working.

One issue regarding the Ellie links is that we don't import the CSS, so the Buttons and Alerts don't have the necessary styling. One approach I was thinking of (at least on stateless and message-less files) is to generate a new function at the bottom of the file which we'll expose:

main : Html Msg
main model =
    div []
      [ Ant.Css.defaultStyles
      , example 
      ]

but I'll focus on the functionality of the URL generator first, before dealing with this.

[supermacro]

Reading through your PR now. Very excited about this 🤩 👏 🤩 👏 🤩 👏