Skip to content

Commit

Permalink
chore: store the daily/monthly filter setting in local storage for cl…
Browse files Browse the repository at this point in the history
…uster historical usage page [DET-5194] (#2161)

* chore: store the daily/monthly filter setting in local storage for cluster historical usage page [DET-5194]

* doc updates
  • Loading branch information
apizzini authored Apr 5, 2021
1 parent 9e3c69d commit 255fa74
Show file tree
Hide file tree
Showing 3 changed files with 50 additions and 28 deletions.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
59 changes: 37 additions & 22 deletions docs/reference/historical-cluster-usage-data.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,25 +4,40 @@
Historical Cluster Usage Data
###############################

- Our goal is to give users insights on how their Determined cluster is
used. To do so, we provide two features: Web UI visualizations for a
quick snapshot of usage and API endpoints to download their resource
allocation data for their own analysis.

- Resource allocation is measured in the number of GPU hours allocated
by Determined. This has two important limitations. Importantly, this
is not resource utilization, so if a user gets 1 GPU allocated but
only utilizes 20% of the GPU, we would still report one GPU hour.
Also, this does not include time the GPU is idle (e.g., time waiting
for a GPU to spin up, or when a GPU is sitting idle and not
deallocated yet). For that reason GPU hours reported by Determined
may be less than GPU hours reported by the cloud.

- Short explanation of the APIs and download CSV button. (Unless all
the API info is already documented via Swagger?)

- Our data is aggregated by Determined metadata (e.g., label, user).
This aggregation is performed nightly, so any data visualized on the
web UI or downloaded via the endpoint is fresh as of the last night.
It will not reflect changes to the metadata of a previously run
experiment (e.g., labels) until the next nightly aggregation.
Our goal is to give users insights on how their Determined cluster is
used. To do so, we provide two features: Web UI visualizations for a
quick snapshot of usage and API endpoints to download their resource
allocation data for their own analysis.

Resource allocation is measured in the number of GPU hours allocated by
Determined. This has two important limitations. Importantly, this is not
resource utilization, so if a user gets 1 GPU allocated but only
utilizes 20% of the GPU, we would still report one GPU hour. Also, this
does not include time the GPU is idle (e.g., time waiting for a GPU to
spin up, or when a GPU is sitting idle and not deallocated yet). For
that reason GPU hours reported by Determined may be less than GPU hours
reported by the cloud.

Our data is aggregated by Determined metadata (e.g., label, user). This
aggregation is performed nightly, so any data visualized on the web UI
or downloaded via the endpoint is fresh as of the last night. It will
not reflect changes to the metadata of a previously run experiment
(e.g., labels) until the next nightly aggregation.

.. image:: /assets/images/historical-cluster-usage-data.png
:width: 100%

########################
Command-line Interface
########################

Historical cluster usage data are accessible through CLI:

- ``det resources raw <start time> <end time>``: get raw allocation
information, where the times are full times in the format
yyyy-mm-ddThh:mm:ssZ.

- ``det resources aggregated <start date> <end date>``: get aggregated
allocation information, where the dates are in the format yyyy-mm-dd.

See :ref:`the CLI reference page <cli>`.
19 changes: 13 additions & 6 deletions webui/react/src/pages/Cluster/ClusterHistoricalUsage.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { useHistory } from 'react-router';

import Section from 'components/Section';
import useResize from 'hooks/useResize';
import useStorage from 'hooks/useStorage';
import { getResourceAllocationAggregated } from 'services/api';

import css from './ClusterHistoricalUsage.module.scss';
Expand All @@ -25,17 +26,22 @@ export enum GroupBy {
Month = 'month',
}

const STORAGE_PATH = 'cluster/historical-usage';
const STORAGE_GROUP_BY_KEY = 'group-by';

const ClusterHistoricalUsage: React.FC = () => {
const [ chartSeries, setChartSeries ] = useState<ResourceAllocationChartSeries>();
const [ filters, setFilters ] = useState<ClusterHistoricalUsageFiltersInterface>({
afterDate: dayjs().subtract(1 + DEFAULT_RANGE_DAY, 'day'),
beforeDate: dayjs().subtract(1, 'day'),
groupBy: GroupBy.Day,
});
const [ isCsvModalVisible, setIsCsvModalVisible ] = useState<boolean>(false);
const [ isUrlParsed, setIsUrlParsed ] = useState<boolean>(false);
const filterBarRef = useRef<HTMLDivElement>(null);
const history = useHistory();
const storage = useStorage(STORAGE_PATH);

const [ filters, setFilters ] = useState<ClusterHistoricalUsageFiltersInterface>({
afterDate: dayjs().subtract(1 + DEFAULT_RANGE_DAY, 'day'),
beforeDate: dayjs().subtract(1, 'day'),
groupBy: storage.getWithDefault(STORAGE_GROUP_BY_KEY, GroupBy.Day),
});

/*
* When filters changes update the page URL.
Expand All @@ -54,9 +60,10 @@ const ClusterHistoricalUsage: React.FC = () => {

// group-by
searchParams.append('group-by', filters.groupBy);
storage.set(STORAGE_GROUP_BY_KEY, filters.groupBy);

history.push('/cluster/historical-usage?' + searchParams.toString());
}, [ filters, history, isUrlParsed ]);
}, [ filters, history, isUrlParsed, storage ]);

/*
* On first load: if filters are specified in URL, override default.
Expand Down

0 comments on commit 255fa74

Please sign in to comment.