Create a web app for continuous evaluation and monitoring of explcheck results

[Witiko]

To provide a sustainable, user-friendly platform for evaluating and showcasing the results of explcheck on all expl3 packages (from both current and historical TeX Live distributions), I propose developing a publicly accessible web app. This app will be updated automatically and offer maintainers, contributors, and the wider TeX community real-time insights into explcheck findings and expl3 usage trends.

Background

In Section 2.2.3 of the project proposal, I discussed conducting a large-scale validation of explcheck by testing all expl3 packages from current and historical TeX Live distributions. The results would inform a TUGboat article, focusing on both quantitative trends in expl3 usage and qualitative insights for future improvements.

Rather than create and discard a one-off experiment, we can build a sustainable solution—a web app—that:

1. Provides a continuous overview of explcheck results,
2. Helps package maintainers track issues without installing or running explcheck locally, and
3. Enables the community of expl3 programmers to explore and learn from aggregated symbol data.

Rationale

1. Easy monitoring: Package maintainers can instantly see potential issues and errors flagged by explcheck without installing or configuring the tool.
2. Continuous evaluation: Monitoring results as we develop explcheck allows us to catch regressions or new issues early, rather than discovering them at the end of the project.
3. Practical example: Demonstrates how explcheck can be used from third-party Lua code.

In the future, the web app could also aggregate all public expl3 symbols from various packages. Similar to how web apps like DeTeXify help authors discover LaTeX2e commands and packages for mathematical symbols, our app could serve as a central repository of expl3 functions, allowing developers to easily discover symbols from different expl3 packages.

Requirements

A pull request (PR) addressing this issue should implement the following changes:

1. Static website with daily updates:

   • Generated by GitHub Actions, published via GitHub Pages at https://witiko.github.io/expltools/explcheck/.
   • All linter results are precomputed and embedded into the site at build time, avoiding the need for dynamic data retrieval. This allows the entire site to be packaged as a single artifact, making it easy to review in pull requests before merging.

2. Summary visualization on landing page:

   • Shows a high-level view of how many expl3 files have been found and the total number of detected warnings/errors.
   • Include a time-series or historical perspective (similar to a bar chart from my recent blog post).
   • Prominently display a search bar.

3. Search and faceted filtering:

   • Selecting the search bar displays a (paginated) list of all found expl3 files and:
     • Allows full-text filename search of expl3 files.
     • Allows faceted search based on the types or categories of detected issues.
     • Dynamically updates the visualization to reflect filtered search results.

4. Detailed file view:

   • Selecting a specific file (or group of files) shows:
     • A summary visualization scoped to that selection.
     • A list of detected issues with relevant code snippets for each.
     • The ability to switch between different TeX Live distributions and compare results over time.
   • The detailed file view should have a stable URL derived from the filename(s) of the expl3 files.

Next steps

1. Start small:
   • First generate a static HTML table of found expl3 files using GitHub Actions.
2. Iterate:
   • Next, generate detailed file views for individual files, potentially as separate HTML files, and link them from the table.
   • Add a stable JSON data source and use it to generate visualizations using e.g. the library D3.js.
3. Hand over to a frontend developer:
   • Provide them with your JSON data structures, the mockups from the previous two steps, and the requirements.
   • Ask them to implement searching, filtering, and other features that are missing at this point.

[Witiko]

@hansonchar: Why not make use of Github actions (instead of building a webapp) to continuously perform regression tests whenever there is any change made to the repository?

That’s a great suggestion—indeed, we already leverage GitHub Actions to run regression tests on known issue areas. However, validating all expl3 packages in TeX distributions adds another layer of insight by helping us discover unknown issues and trends that might slip through standard regression tests. With this approach, we can monitor changes across packages, identify unexpected patterns, and maintain a more holistic view of overall project health and progress. However, unlike regression tests, this approach requires a human in the loop, so we need a human-computer interface—a web app.

[Witiko]

I spent the weekend pondering this ticket and I think we can significantly simplify the requirements from the original post without losing anything important. Namely:

4. Detailed file view:
   • Selecting a specific file (or group of files) shows:
     • [...]
     • The ability to switch between different TeX Live distributions and compare results over time.

As outlined in the original post and in #28 (comment), two main use cases are envisioned for the web app: 1) continuous evaluation with a broader dataset compared to our regression tests and 2) exploration of issues (and symbols) in expl3 packages by package developers and maintainers.

For the first use case, we don't necessarily need a web app. We can continuously run explcheck on all packages in historical TeX Live Docker images and compare the results to the expected results recorded in the repository. This is automatic and would essentially function as an extended set of regression tests, as suggested by @hansonchar.

For the second use case, we don't necessarily need the ability to switch between different TeX Live distributions. Instead, it seems more useful to display as current result as possible to provide an illusion of a real-time service even though the app would essentially be only a static website with frequent updates.

Therefore, we don't to generate the web app from multiple TeX Live distributions or the ability to switch between different TeX Live distributions within the app. This should simplify the data processing pipeline and also eliminate the following requirement:

2. Summary visualization on landing page:
   • [...]
   • Include a time-series or historical perspective (similar to a bar chart from my recent blog post).

If we eliminate the historical perspective, it seems that we can get rid of the visualization part altogether. This eliminates the following requirements:

2. Summary visualization on landing page:
   • Shows a high-level view of how many expl3 files have been found and the total number of detected warnings/errors.
3. Search and faceted filtering:
   • [...]
   • Dynamically updates the visualization to reflect filtered search results.
4. Detailed file view:
   • Selecting a specific file (or group of files) shows:
     • A summary visualization scoped to that selection.

This leaves us with two smaller tasks (1, 2) for now and another task (3) for the future:

1. Add a continuous integration (CI) step that:
   1. Processes all expl3 files in a historical release of TeX Live.
   2. Compares the results to the expected results recorded in the repository.
   3. Fails if there are any differences and displays them in the CI log.
2. Add a continuous integration (CI) step that:
   1. Generates a static web site with two levels:
      1. A list of all expl3 files from the current release of TeX Live, which allows full-text filename search and faceted search based on the types of detected issues.
      2. A detailed file view with a persistent URL, which shows a list of detected issues with relevant code snippets for each.
   2. Uploads the web site as an artifact.
   3. For CI on the main branch, publishes the web site on GitHub Pages.
3. After we have implemented semantic analysis, generate a web library of expl3 functions. Instead of files and issues, this web site would display a list of detected expl3 symbols, allow full-text search by name and type, and display a detailed view with a list of expl3 files that use a symbol.

[Witiko]

I created separate tickets #31, #32, and #33 to track the sub-issues outlined in #28 (comment).