Nice! Read the few sections below to understand how this project is structured and how to implement new features.
This section explore some topics which explain globally how metrics was designed and how it works.
💬 Creating SVGs images on-the-fly
Metrics actually exploit the possibility of integrating HTML and CSS into SVGs, so basically creating these images is as simple as designing static web pages. It can even handle animations and transparency.
SVGs are templated through EJS framework to make the whole rendering process easier thanks to variables, conditional and loop statements. Only drawback is that it tends to make syntax coloration a bit confused because templates are often misinterpreted as HTML tags markers (<%= "EJS templating syntax" %>
).
Images (and custom fonts) are encoded into base64 to prevent cross-origin requests, while also removing any external dependencies, although it tends to increase files sizes.
Since SVG renders differently depending on OS and browsers (system fonts, CSS support, ...), it's pretty hard to compute dynamically height. Previously, it was computed with ugly formulas, but as it wasn't scaling really well (especially since the introduction of variable content length plugins). It was often resulting in large empty blank spaces or really badly cropped image.
To solve this, metrics now spawns a puppeteer instance and directly render SVG in a browser environment (with all animations disabled). An hidden "marker" element is placed at the end of the image, and is used to resize image through its Y-offset.
Additional bonus of using pupeeter is that it can take screenshots, making it easy to convert SVGs to PNG output.
Finally, SVGs image can be optimized through svgo, which helps to remove unused attributes and blank space, while also reducing a bit the file size.
💬 Gathering external data from GitHub APIs and Third-Party services
Metrics mostly use GitHub APIs since it is its primary target. Most of the time, data are retrieved through GraphQL to save APIs requests, but it sometimes fallback on REST for other features. Octokit SDKs are used to make it easier.
As for other external services (Twitter, Spotify, PageSpeed, ...), metrics use their respective APIs, usually making https requests through axios and by following their documentation. It would be overkill to install entire SDKs for these since plugins rarely uses more than 2/3 calls.
In last resort, pupeeter is seldom used to scrap websites, though its use tends to make things slow and unstable (as it'll break upon HTML structural changes).
💬 Web instance and GitHub action similarities
Historically, metrics used to be only a web service without any customization possible. The single input was a GitHub username, and was composed of what is now base
content (along with languages
and followup
plugin, which is why they can be computed without any additional queries). That's why base
content is handled a bit differently from plugins.
As it gathered more and more plugins over time, generating a single user's metrics was becoming costly both in terms of resources but also in APIs requests. It was thus decided to switch to GitHub Action. At first, it was just a way to explore possibilities of this GitHub feature, but now it's basically the full-experience of metrics (unless you use your own self-hosted instance).
Both web instance and Action actually use the same entrypoint so they basically have the same features. Action just format inputs into a query-like object (similarly to when url params are parsed by web instance), from which metrics compute the rendered image. It also makes testing easier, as test cases can be reused since only inputs differs.
💬 Testing and mocking
Testing is done through jest framework.
While the best would be to work with real data during testing, to avoid consuming too much APIs requests for testing (and to be more planet friendly), they're mocked using JavaScript Proxies and Faker.js. Basically function calls are "trapped" and send randomly generated data from Faker.js if we're in a development environment.
This section explain how to create new templates and plugins, and contains references about used packages and project structure. It also lists which contributions on main repository are accepted.
🤝 Accepted contributions
Thanks for wanting to help metrics growing!
Review below which contributions are accepted:
Section | Examples | Addition | Editions |
---|---|---|---|
🧩 Plugins | ✔️ | ✔️ | |
🖼️ Templates | ❌ | ⭕ | |
🧪 Tests | tests/metrics.test.js |
✔️ | ✔️ |
🧱 Core | app/metrics/ , Dockerfile , package.json ... |
❌ | ⭕ |
🗃️ Repository | .github/ , LICENSE , CONTRIBUTING.md , ... |
❌ | ❌ |
Legend
- ✔️: Contributions welcomed!
- ⭕: Contributions welcomed, but must be discussed first with a maintainer
- ❌: Only maintainers can manage these files
Before working on something, ensure that it isn't listed in In progress and that no open pull requests (including drafts) already implement what you want to do.
If it's listed in Roadmap and todos be sure to let maintainers that you're working on it. As metrics remains a side project, things being working on can change from one day to another.
If you're unsure, always open an issue to obtain insights and feedback 🙂
And even if your changes don't get merged in lowlighter/metrics, please don't be too sad. Metrics is designed to be highly customizable, so you can always decide to generate metrics on your forked repository!
🗂️ Project structure
This section explain how metrics is structured.
source/app/metrics/
contains core metrics filessource/app/action/
contains GitHub action filesindex.mjs
contains GitHub action entry pointaction.yml
contains GitHub action descriptor
source/app/web/
contains web instance filesindex.mjs
contains web instance entry pointinstance.mjs
contains web instance source codesettings.example.json
contains web instance settings examplestatics/
contains web instance static filesapp.js
contains web instance client source codeapp.placeholder.js
contains web instance placeholder mocked data
source/app/mocks/
contains mocked data filesapi/
contains mocked api dataaxios/
contains external REST APIs mocked datagithub/
contains mocked GitHub api data
index.mjs
contains mockers
source/plugins/
contains source code of pluginsREADME.md
contains plugin documentationmetadata.yml
contains plugin metadataindex.mjs
contains plugin source codequeries/
contains plugin GraphQL queries
source/templates/
contains templates filesREADME.md
contains template documentationimage.svg
contains template image used to render metricsstyle.css
contains style used to render metricsfonts.css
contains additional fonts used to render metricstemplate.mjs
contains template source code
tests/
contains testsmetrics.test.js
contains metrics testers
Dockerfile
contains docker instructions used to build metrics imagepackage.json
contains dependencies and command line aliases
📦 Packages
Below is a list of used packages.
- express/express.js and expressjs/compression
- To serve, compute and render a GitHub user's metrics
- nfriedly/express-rate-limit
- To apply rate limiting on server and avoid spams and hitting GitHub API's own rate limit
- octokit/graphql.js and octokit/rest.js
- To perform request to GitHub GraphQL API and GitHub REST API
- mde/ejs
- To render SVG images
- ptarjan/node-cache
- To cache generated content
- renanbastos93/image-to-base64
- To generate base64 representation of users' avatars
- svg/svgo
- To optimize generated SVG
- axios/axios
- To make HTTP/S requests
- actions/toolkit
- To build the GitHub Action
- vuejs/vue and egoist/vue-prism-component + prismjs/prism
- To display server application
- puppeteer/puppeteer
- To scrap the web
- libxmljs/libxmljs
- To test and verify SVG validity
- facebook/jest and nodeca/js-yaml
- For unit testing
- marak/faker.js
- For mocking data
- steveukx/git-js
- For simple git operations
🖼️ Templates
Templates requires you to be comfortable with HTML, CSS and JavaScript (EJS flavored).
Metrics does not really accept contributions on default templates in order to avoid bloating main repository with a lot of templates, but fear not! Users will still be able to use your custom templates thanks to community templates!
If you make something awesome, don't hesistate to share it!
For a quick start, use:
npm run quickstart -- template <template_name>
💬 Creating a new template from scratch
Find a cool name for your template and create an eponym folder in source/templates
.
Then, you'll need to create the following files:
README.md
will contain template description and documentationimage.svg
will contain the base render structure of your templatepartials/
is a folder that'll contain parts of your template (called "partials")partials/_.json
is a JSON array which lists your partials (these will be displayed in the same order as listed, unless if overriden by user withconfig_order
option)
The following files are optional:
fonts.css
can contain your custom fonts (base64 encoded) if neededstyles.css
can contain your CSS that'll style your templatetemplate.mjs
can contain additional data processing and formatting at template-level
Optional files will fallback to the one defined in classic
template if unexistant.
Note that by default, template.mjs
is skipped when using official release with community templates, to prevent malicious code to leaks token and credentials.
💬 Creating a README.md
Your README.md
will document your template and explain how it works.
It must contain at least the following:
### 📕 My custom template
<table>
<td align="center">
<img src="">
<img width="900" height="1" alt="">
</td>
</table>
#### ℹ️ Examples workflows
'''yaml
- uses: lowlighter/metrics@latest
with:
# ... other options
setup_community_templates: user/metrics@master:template
template: "@template"
'''
💬 Creating image.svg
Once you finished setting up template folder structure, paste the following in image.svg
to get started:
<svg xmlns="http://www.w3.org/2000/svg" width="480" height="99999" class="<%= !animated ? 'no-animations' : '' %>">
<defs><style><%= fonts %></style></defs>
<style><%= style %></style>
<foreignObject x="0" y="0" width="100%" height="100%">
<div xmlns="http://www.w3.org/1999/xhtml" xmlns:xlink="http://www.w3.org/1999/xlink">
<% for (const partial of [...partials]) { %>
<%- await include(`partials/${partial}.ejs`) %>
<% } %>
<div id="metrics-end"></div>
</div>
</foreignObject>
</svg>
Let's explain what it does.
fonts
and style
variables will be populated with the same content as your fonts.css
and styles.css
files.
Like said previously, if these does not exists, it'll contain the same content as the classic
template files.
The main loop will iterate on partials
variable which contains your partials index set in _.json
.
Finally, you may have noticed that height
is set to a very high number, and that there is a #metrics-end
element at the bottom of the SVG template. This is because rendered height is computed dynamically through a puppeteer browser instance which locate #metrics-end
and use its y-coordinate and config_padding
to set final height. So you should leave it like this to ensure your rendered image will be correctly sized.
💬 Customizing templates with partials
Partials are sections that'll be displayed in rendered metrics.
It's just HTML with CSS which can be templated through EJS framework.
Basically, you can use JavaScript statements in templating tags (<% %>
) to display variables content and to programmatically create content.
💬 Using custom fonts
This is actually not recommended because it drastically increases the size of generated metrics, but it should also make your rendering more consistant. The trick is to actually restrict the charset used to keep file size small.
Below is a simplified process on how to generate base64 encoded fonts to use in metrics:
-
- Find a font on fonts.google.com
- Select regular, bold, italic and bold+italic fonts
- Open
embed
tab and extract thehref
-
- Open extracted
href
and append&text=
params with used characters from SVG
- e.g.
&text=%26%27"%7C%60%5E%40°%3F!%23%24%25()*%2B%2C-.%2F0123456789%3A%3B<%3D>ABCDEFGHIJKLMNOPQRSTUVWXYZ%5B%5D_abcdefghijklmnopqrstuvwxyz%7B%7D~─└├▇□✕
- Open extracted
-
- Download each font file from url links from the generated stylesheet
-
- Convert them into base64 with
woff
extension on [transfonter.org]https://transfonter.org/) and download archive
- Convert them into base64 with
-
- Extract archive and copy the content of the generated stylesheet to
fonts.css
- Extract archive and copy the content of the generated stylesheet to
-
- Update your template
- Include
<defs><style><%= fonts %></style></defs>
to yourimage.svg
- Edit your
style.css
to use yout new font
🧩 Plugins
Plugins are self-sufficient and independant code functions that gather additional data from GitHub APIs or external sources.
💬 Plugin guidelines
- A plugin should never be dependent on others plugins
- But they're allowed to use data gathered by main metrics function
- Avoid the need of new external dependencies (like SDKs)
- Most of the time, SDKs are overkill when a few HTTP calls do the trick
imports
probably contains a library that can help you achieving what you want
- Avoid using raw command when possible (like spawning sub-process)
- Sub-process should be platform agnostic (i.e. working on most OS)
- Errors should always be handled gracefully by displaying an error message when it fails
- When possible, try to display explicit error messages
For a quick start, use:
npm run quickstart -- plugin <plugin_name>
💬 Creating a new plugin
Find a cool word to name your plugin and create an eponym folder in source/plugins
folder.
You'll also need to find an unused emoji that you'll be able to use as your plugin icon.
Then create an index.mjs
in your plugin folder and paste the following code:
//Setup
export default async function ({login, q, imports, data, computed, rest, graphql, queries, account}, {enabled = false} = {}) {
//Plugin execution
try {
//Check if plugin is enabled and requirements are met
if ((!enabled)||(!q/* your plugin name */))
return null
//Results
return {}
}
//Handle errors
catch (error) {
throw {error:{message:"An error occured", instance:error}}
}
}
The following inputs are available:
login
is set to GitHub loginq
contains all query parametersimports
contains libraries and utilitaries that are shared amongst pluginsimports.url
refers to NodeJSurl
libraryimports.os
refers to NodeJSos
libraryimports.fs
refers to NodeJSfs
libraryimports.paths
refers to NodeJSpaths
libraryimports.util
refers to NodeJSutil
libraryimports.imgb64
refers to renanbastos93/image-to-base64imports.axios
refers to axios/axiosimports.puppeteer
refers to puppeteer/puppeteerimports.run
is an helper to run raw commandimports.shuffle
is an helper to shuffle arrayimports.__module
is an helper to find__dirname
from a module url- And more...
data
andcomputed
contains all data (and computed data) gathered by various APIs from main metrics functiongraphql
andrest
contains octokit clients for GitHub APIqueries
contains autoloaded GraphQL queries with replacersaccount
contains the type of account being worked on ("user" or "organization")
The second input contains configuration settings from settings.json, which is mostly used by web instances.
Content of these parameters should never be edited directly, as your plugin should only return a new result.
Plugins are autoloaded so you do not need to do anything special to register them.
💬 Gathering new data from GitHub APIs and from Third-Party services
For GitHub related data, always try to use their GraphQL API or their REST API when possible. Use puppeteer
in last resort.
When using GraphQL API, queries
object autoloads queries from your plugin queries
directory and will replace all strings prefixed by a dollar sign ($
) with eponym variables.
For example:
//Calling this
await graphql(queries.myquery({login:"github-user", account:"user"}))
//With this in source/queries/myquery.graphql
query MyQuery {
$account(login: "$login") {
name
}
}
//Will have the same result as calling this
await graphql(`
query MyQuery {
user(login: "github-user") {
name
}
}
`)
For REST API, check out their documentation.
As for Third-Party services, always prefer using their APIs (you can use imports.axios
for easy HTTP requests) when they exists before having recourse to imports.puppeteer
.
New external dependencies should be avoided at all costs, especially since most of the time it's overkill to setup a new SDK.
💬 Creating a partial to display your plugin result
Create new files in partials
of source/templates
you want to support with .ejs
extension.
You can paste the following for a quick start:
<% if (plugins./* your plugin name */) { %>
<section>
<div class="row">
<% if (plugins./* your plugin name */.error) { %>
<section>
<div class="field error">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 16 16" width="16" height="16"><path fill-rule="evenodd" d="M2.343 13.657A8 8 0 1113.657 2.343 8 8 0 012.343 13.657zM6.03 4.97a.75.75 0 00-1.06 1.06L6.94 8 4.97 9.97a.75.75 0 101.06 1.06L8 9.06l1.97 1.97a.75.75 0 101.06-1.06L9.06 8l1.97-1.97a.75.75 0 10-1.06-1.06L8 6.94 6.03 4.97z"></path></svg>
<%= plugins./* your plugin name */.error.message %>
</div>
</section>
<% } else { %>
<section>
<%# Do stuff in there -%>
</section>
<% } %>
</div>
</section>
<% } %>
Let's explain what it does.
First conditional statement will ensure that your partial only execute when your plugin is enabled.
The nested one will check if your plugin resulted in an error, and if that's the case, it'll display an error message instead. Else, if it's successful, you'll get the second section in render.
Plugins errors should always be handled gracefully when possible.
If you need additional CSS rules, edits the style.css
of edited template.
💬 Fast prototyping with web instance
The easiest way to test and prototype your plugin is to use a web instance.
Configure a settings.json with a valid GitHub token and with debug mode enabled.
Then start a web instance with npm start
(you may have to run npm install
if that's the first time you use the web instance).
Then try to generate your metrics in your browser with your GitHub user and your plugin enabled, and see if it works as expected:
http://localhost:3000/your-github-login?base=0&your-plugin-name=1
💬 Registering plugin options in metadata.yml
metadata.yml
is a special file that will be used to parse user inputs and to generate final action.yml
name: "🧩 Your plugin name"
# Estimate of how many GitHub requests will be used
cost: N/A
# Supported modes
supports:
- user
- organization
- repository
# Inputs list
inputs:
# Enable or disable plugin
plugin_custom:
description: Your custom plugin
type: boolean
default: no
The following types are supported:
string:
type: string
select:
type: string
values:
- allowed-value-1
- allowed-value-2
- ...
boolean:
type: boolean
number:
type: number
ranged:
type: number
min: 0
max: 100
array:
type: array
format: comma-separated
array_select:
type: array
format: comma-separated
values:
- allowed-value-1
- allowed-value-2
- ...
json:
type: json
💬 Create mocked data and tests
Creating tests for your plugin ensure that external changes don't break it.
You can define your tests cases in tests.yml
in your plugin directory, which will automatically test your plugin with:
- Metrics action
- Metrics web instance
- Metrics web instance placeholder (rendered by browser)
As most of APIs (including GitHub) usually have a rate-limit to ensure quality of their service. To bypass these restrictions but still perform tests, you must mock their data which simulates APIs call returns.
Add them in source/app/mocks/api/
folder.
If you're using axios
or GitHub GraphQL API, these files are autoloaded so you just need to create new functions (see other mocked data for examples).
If you're using GitHub REST API, add your mocks in source/app/mocks/rest
with same path as octokit.
💬 Creating a README.md
Your README.md
will document your plugin and explain how it works.
It must contain at least the following:
### 🧩 Your plugin name
<table>
<td align="center">
<img src="">
<img width="900" height="1" alt="">
</td>
</table>
#### ℹ️ Examples workflows
[➡️ Available options for this plugin](metadata.yml)
'''yaml
- uses: lowlighter/metrics@latest
with:
# ... other options
plugin_custom: yes
'''
Note that you must keep <table>
tags as these will be extracted to autogenerated global README.md
with your example.
Written by lowlighter