Explorers will help you to compare 1000s of AI experiments with a few clicks. Explorers are the main tools that Aim is built around.
In this section we will go through the Aim explorers, introduce their features and how to use them.
Use Metrics explorer to search and compare 1000s of ML training metrics.
The Metrics Explorer allows you to search, group and compare your metrics. Due to this and number of other visual features on the Metrics Explorer, you will save considerable amounts of time when comparing experiments- compared to other open-source experiment tracking tools.
The Metric Explorer has the following main sections:
- Metrics Select: to select the metrics for exploration
- Search bar: to query the runs for exploration
- Charts explorer: the space where the metrics are rendered
- Metrics modifiers: all the grouping, chart division and other metrics modifier tools
- Context table: the full information about the selected metrics is available
.. note:: There is also an advanced mode of search is available too where you can use the full Aim QL (more on this further in this section).
There are two ways you can query metrics and runs
An overview of what you can do with queried metrics - the modifiers:
- Group by tracked parameters
- Aggregate grouped metrics
- Configure axes properties (x-axis alignment, axes range)
- Change scale of the axes (linear or log)
- Apply smoothing
- Ignore outliers
- Metric highlight modes (metric on hover, run on hover)
- Set chart tooltip parameters
- Apply zoom in/out on charts
- Export chart as image
On the Metrics Explorer, there is + Metrics
button.
Once pressed, a dropdown will appear with all your tracked metrics and their contexts flattened.
The dropdown is searchable - so you can get to your metric of interest within a stroke!
The Search bar is located below the + Metric
button. It allows to do a pythonic query (that is eval-ed as python statement) over every param you have tracked.
Search runs with Aim QL
Once you press the Enable advanced search mode
button underneath the main Search
button, it will enable the full Aim QL search editor - to query the metrics, the runs via full Aim QL
Here is an example:
((metric.name == 'bleu' and metric.context.subset == 'val') or (metric.name == 'loss' and metric.context.subset == 'train')) and 1e-5 < run.hparams.learning_rate < 1e-2
Grouping selected metrics by any tracked params will allow you to quickly distinguish the most impactful params, decisions you have made (the preprocessing steps, the hyperparams etc).
The parameters include not only the ones you have tracked but also the native Aim objects too such as
metric.name
metric.context.[context_key]
run.hash
There are several ways you can group the selected metrics and runs - by color, by stroke and by chart.
Use this to divide the selected metrics into different clusters based on selected values of params. Each cluster gets colored differently.
There are a number of options available when grouping
- group by values - divides into clusters as per the values of selected params)
- reverse grouping - divides into clusters by every param except for the chosen one.
The grouping colors are picked randomly, however it is possible to fix with the advanced coloring features.
Here are the features in the advanced mode:
- Fix the colors of the grouping
- Control the color palette to use during the grouping
Groups the metrics by a stroke style. Has all the rest of the other features available on the color grouping except the advanced mode.
The end result of using this feature: divides into subplots based on the value of the selected params. Why this is a grouping mechanism? It groups the metrics belonging to the same group into separate charts.
The metrics aggregation helps to quickly see the trends of each individual group of metrics. See more about metrics grouping.
There are two aspects of aggregation you can control:
- the trend-line
- the area that the group of metricsc take
The trend-line calculation methods:
- Mean
- Median
- Min
- Max
The area calcualtion methods:
- None (when you'd like to remove the area)
- Min/Max
- Mean +/- Standard Deviation
- Mean +/- Standard Error
- Confidence Interval (95%)
Pls see the screenshot:
Axes properties section is for aligning metrics by time, epoch or another metric and for setting axes range manually.
Following types of metrics alignment are available: Step, Epoch, Relative Time, Absolute Time and Custom Metric. By default, metrics aligned by Step.
By setting metrics alignment to Step, x-axis values will represent the steps of tracked metrics.
By setting metrics alignment to Epoch, x-axis values will represent the epochs of tracked metrics.
By setting metrics alignment to Relative Time, x-axis values will represent by HH:mm:ss
, duration of tracking process.
By setting metrics alignment to Absolute Time, x-axis values will represent by date HH:mm:ss D MMM, YY
, since the start date of the first run until the last run.
By setting metrics alignment to Custom Metric, x-axis values will represent selected metric values, you can detect correlations between queried metrics and selected metric.
To fix an axis range across all the charts, set the corresponding axis minimum and maximum bounds in the form.
Axes Scale section gives ability to display axes scale's linear or logarithmic.
By default, axes scale's are Linear.
While smoothing the chart, the data points are modified so individual points higher than the adjacent points (presumably because of noise) are reduced, and points that are lower than the adjacent points are increased leading to a smoother signal. You can select curve interpolation methods: Linear or Cubic. By default, chart smoothing in Exponential moving average mode and curve interpolation method is Linear.
An exponential moving average, also known as an exponentially weighted moving average (EWMA), is a first-order infinite impulse response filter that applies weighting factors which decrease exponentially.
When you center the moving averages, the data points are placed at the center of the range rather than the end of it. This is done to position the moving average values at their central positions in time.
An outlier is an observation that lies an abnormal distance from other values in a random sample from a population. Examination of the data for unusual observations that are far removed from the mass of data. These points are often referred to as outliers.
Excluding outliers can cause your results to become statistically significant. By default, outliers are ignored.
Highlighting functionality is useful for filtering metrics and highlight only hovered metric. Following types of highlighting mode are available: Highlight Off, Highlight Metric on Hover, and Highlight Run on Hover. By default, highlighting mode is the Highlight Run on Hover.
By setting Highlight mode Off, there is no highlighting functionality on hover.
By setting Highlight mode Metric on Hover, mouse point closest metric highlights and other metrics displays with opacity.
By setting Highlight mode Run on Hover, mouse point closest metric highlights and highlighted metric corresponding run also highlights other metrics displays with opacity.
You can select tooltip parameters to show params and values in tooltip Params section. You can select hide or show button to display or hide tooltip on hover.
Metric explorer also, gives ability to export your chart as image.
By clicking export button
from control panel, will be opened chart preview modal.
You can change exportable chart image width
, single chart height
, set image name
and format
.
Following formats of chart export are available: SVG
, JPEG
, PNG
.
Track intermediate images search easily by using select form functional compare them on the Images Explorer by using reach controls panel.
Features:
- Easily query any image
- Group images by run parameters, step and index
- Use controls from right control panel to configure workspace
Use select form to easily query any image. There are two option to query images using dropdown, by using Aim QL language and advance mode for Aim QL.
- Click on Images button
- Select options you are want to use in query
- Click on the Search button
- Click on pencel icon in the right side of select form to show input
- Type advance Aim QL query
- Click on the Search button
Use select grouping dropdown which is located in the right top corner of the image explore page.
- Click on grouping button
- Select fields by which you want to groupe images
Grouping will be apply after each field selection also you can select grouping mode (Group or Reverse)
Any change in controls will help to explore images better on the workspace
-
Click on image property button
-
Select value from dropdown to align image. (by default dropdown value is
Height
). Use slider to configure value for scale relative to window size by default scale value is15%
. -
Use image rendering variation by default value of this control is
Pixelated
- Click on image sorting button
- Select fields for sorting images. Selection ordering is meaningful and data will be sorting by selection order. Bellow is visible Ordered By list where contains all selected fields from dropdown. You can remove any already selected field by clicking on
x
icon or change sorting direction by clicking radio button Asc or Desc. Default selected direction is Asc. - For reset all existing sorting fields you can simply click on Reset Sorting button
You can select tooltip parameters to show params and values in tooltip Params section. You can select hide or show button to display or hide tooltip on hover.
Params explorer helps you to represent high dimensional data as a multi-dimensional visualization. Features:
- Easily query any metrics and params
- Group runs by color, stroke, or chart
- Make the crossings easier with curve interpolation
- Learn patterns and colorations easier colored by the last dimension with a color indicator
Select params and metrics from dropdown
Search runs with Aim QL
Group by color, stroke, or chart with selected parameters
By clicking on the Curve interpolation button in the Controls panel, it's possible to make lines from straight to curve to show correlations between non-adjacent axes.
By clicking on the Color indicator button in the Controls panel, it's possible to turn on lines gradient coloring by the last dimension.
Scatter explorer gives ability to visualize correlations between metric last value data with hyper-parameter.
It represents graph where corresponding values from a set of data are placed as points on a coordinate plane. A relationship between the points is sometimes shown to be positive, negative, strong, or weak.
Abilities provided by Scatter explorer
- Easily align metric last value data with hyper parameter
- Group runs by color and chart
- Represent the points with trend line
- Export chart
Select params and metrics from X
and Y
axes dropdowns to align metric last value data with hyper-parameter.
- X axis
- Y axis
Also, you can search runs with Aim QL
Easily group data by color and chart with selected parameters.
- By
Color
- By
Chart
A trend line is a straight line that best represents the points on a scatter plot
. The trend line may go through some points but need not go through them all.
From trend line options popover you can change regression from Linear
(by default) to LOESS
(locally weighted smoothing), which creates a smooth line through a scatter plot
to help you to see relationship between variables and foresee trends. Also, you can change the bandwidth
with slider
Scatter explorer also, gives ability to export
your chart as image
.
By clicking export button
from control panel, will be opened chart preview modal.
You can change exportable chart image width
, single chart height
, set image name
and format
.
Following image formats are available export: SVG
, JPEG
, PNG
.