Introduce instrumentation report utilities#197
Closed
AlexJones0 wants to merge 11 commits into
Closed
Conversation
Although it can currently be inferred/derived from the full name of the job, having the block (& block variant) information explicitly available in the recorded metadata is much more useful. This means that we can make visualizations and calculations of regression results partitioned by block or block variant. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
Define a common protocol for instrumentation report visualisations with a rendering interface. The idea is to produce a single HTML instrumentation report with various different visualisations (which themselves could be graphs, images, text, etc.) which can be switched between. The rendered outputs themselves are just embedded HTML fragments (or `None` on failure). This should be simple enough whilst allowing the level of extensibility and flexibility that is needed for the current level of instrumentation reporting. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
Add a registry (like the existing registry/factory for the different instrumentation types) that we can use to add new instrumentation visualisations. In the same manner that we currently support plugins to extend the instrumentation functionality of DVSim, this will allow users extending DVSim to hook in and add their own custom instrumentation visualisations as well. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
This will allow reports to add custom content to the HTML head if desired for e.g. styling purposes. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
Whenever a DVSim scheduler run completes and we flush the JSON instrumentation report, add additional logic to create the HTML instrumentation report as well - under the same report directory that is currently being used for the existing simulation HTML report outputs. It is not 100% clear whether this is the best place to integrate the instrumentation reporting yet. It seems like it might be more sensible to do it alongside the simulation reporting to keep common functionality grouped together, but this then means that we do not get the same benefits for other flows (e.g. linting, formal) that do not have their own custom HTML reports. So for now, we keep this in the generic scheduler running logic. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
This commit introduces the instrumentation report template itself, and a function `render_html_report` for rendering the instrumentation report with a list of given instrumentation report visualization implementations. Some key things to note: - We use `plotly.offline` to get the version of the minified plotly.js that is packaged with the plotly Python package. Like vendored static files this will still work in an airgapped environment, but this way it should remain in sync even when the plotly dependency has its version updated. - We only include plotly JS (which is around 4MB minified) as a dependency when we actually have renders that might use it. - For now we include all visualizations as different tabs on the same page. This is to keep things simple as the goal is to keep all the report logic relatively small and self-contained, but this could be expanded upon in the future if it is deemed too restrictive of an interface. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
Intended to be used for the instrumentation reporting logic, but this is general enough that it can just be placed in the time utils. The `format_time_as_hms` function gets a time format like `12h 34m 56.79s`. As far as I could tell, there is no nice way to do this using e.g. the existing datetime standard library, and this is simple to implement manually, so add it as a utility here. The `format_time_metric` is an extension of this intended for the display of time metrics in the instrumentation reports. It reports that time in hours, minutes and seconds, but also puts the time in raw seconds alongside it (e.g. `2h 15m 37.21s (8,137.21s)`). Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
This commit adds some helper methods to the full instrumentation report pydantic model to make generation of instrumentation report visualizations easier. Notably, it introduces a new `ConcreteJobTimingMetrics` model. A common pattern within the instrumentation logic is to only get jobs for which we definitely know the start and end time (and thus the duration), and to discard other jobs. We must keep the partial model since this guarantee does not hold during instrumentation, but we can make a helper that can filter the jobs and return the concrete models. This saves a lot of trouble with static type analysis and the need to constantly re-assert the presence of timing information throughout instrumentation logic. The other `get_run_time_info` is an additional cautious helper method. Generally we would always expect the scheduler timing info to be populated, but in case it is not for whatever reason, we can derive a near approximation for these times by simply examining the start and end times of the jobs themselves. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
The additional kaleido dependency is needed so that plotly is able to render & export graphs to image formats (e.g. PNG) instead of directly rendering some dynamic HTML graph. This is needed for graphs for large runs (> 10K jobs), where the directly rendered graphs grow to enormous sizes if not compressed via conversion to an image. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
Add some utility functions and constant values to assist in instrumentation report visualisation creation. This includes the ability to make (cyclically) repeating colour mappings for datasets with a large number of categories, generic logic to make a hover tooltip that gives information about a job (optionally augmented by the presence of metadata), and a helper for rendering large figures which dynamically switches to rendering a PNG based on whether a threshold number of displayed "entities" (points / bars / etc.) are being rendered. Note that for now we just embed the PNG as a base64 encoded image in the HTML directly. In the future it would probably be a good idea to save these images separately and load them in via HTML src references - but for now this complicates the abstraction interface to be able to handle the edge case of very large graphs, so this is left as a TODO. Also define some useful constants that should remain true across many visualisations: target heights, thresholds & layout/presentation configuration options. Signed-off-by: Alex Jones <alex.jones@lowrisc.org>
Contributor
Author
|
The licensecheck issue causing a CI failure is a minor issue that can be easily worked around (the Specifically:
However it turns out that since Plotly (and many other big Python visualization frameworks) are based in JS, they need a browser to render before being converted to an image. Even worse, Plotly relies on the Rather annoyingly, although Kaleido's README.md mentions the requirement for Chromium, the Plotly README.md says:
(which feels a little misleading). I'd really rather not make DVSim depend on a specific browser being installed if possible. One solution would be to just not render these graphs at scale, but another solution I'm exploring which looks quite feasible is just to render these graphs in Since I'm exploring that avenue and I don't intend to add the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: this PR is marked as a draft as it currently depends on #194 and #196. The first commit of this PR is from #194, and the 2nd-7th commits are from that #196. These can be safely ignored - only the 8th-11th commits are relevant to this PR. When #194 and #196 are merged, this PR will be updated and marked as ready to review, but is otherwise ready for review.
This PR is the seventh of a series of PRs to introduce instrumentation reporting to DVSim.
This PR adds some utility functions & methods that will be used throughout the various instrumentation visualizations/graphs. These are not used yet, in an effort to split the changes up into multiple PRs that should be easier to review. Changes include:
datetime.See the commit messages for more information.