Check this page for a summary of dvvy's capabilities.
Read on for information pertaining to the installation and configuration of the dvvy Chargeback app for Splunk Enterprise, part of the RedFactor Success Suite.
Create an organization identifier ("Org ID") structure for all of the teams, departments, lines of business, etc. This structure should reflect the way you intend to report on utilization and charges. Below is a simple example:
| Group (group_display field in dvvyGroups) | Org ID (group field in dvvyGroups) |
|---|---|
| Network Operations Center (NOC) | noc |
| Security Operations Center (SOC) | soc |
| Application Development | appdev |
| Systems Engineering | syseng |
Once you have defined your Org IDs, add a stanza to your inputs.conf files (and any HEC sources) throughout your environment to create the index-time field (pipeline key). This field will establish data ownership for dvvy dashboards and reports.
Any data source that you want to track in dvvy must have an Org ID assigned. The dvvy app will automatically track data sources that contain the field by way if the KV - Track Data report.
Update forwarder and other inputs.conf configuration on a data source by data source basis with Org ID (see the _meta parameter on line 2 of the sample inputs below).
inputs.conf Example:
| NOC Data Source: | SOC Data Source: |
|---|---|
[monitor:///var/log/syslog] _meta = org_id::noc disabled = false index = linux sourcetype = syslog |
[monitor:///var/log/syslog] _meta = org_id::soc disabled = false index = linux sourcetype = syslog |
The example above creates an index-time field of org_id with values of "noc" and "soc" respectively.
You are not required to use the field name org_id - any field name can be used. If you use a field other than org_id, you will need to update the CUSTOMER_IDENTIFIER macro accordingly. Org IDs can be any alphanumeric value.
The dvvy app is installed on your search tier only (standalone or search head cluster). It relies upon the KV store and summary indexes. It is recommended that you follow Splunk best practices, to include forwarding your summary indexes from your search heads to your indexers.
We recommend the use of the Lookup File Editor app for working with dvvy kv store collections.
All global configuration is stored in the dvvyConfig KV store collection.
Below is an explanation of the configuration parameters:
-
chargeModeLicenseGB: Controls how data ingest (GB volume) charges are calculated. Possible values: simple, over_entitlement
-
chargeModeRTSearchRuntime: Controls how real-time search runtime charges are calculated. Possible values: simple, over_entitlement
-
chargeModeSearchRuntime: Controls how search runtime charges are calculated. Possible values: simple, over_entitlement
-
chargeModeStorageTB: Controls how storage charges are calculated. Possible values: simple, over_entitlement
-
chargeModeVCPU: Controls how VCPU charges are calculated. Possible values: simple, over_entitlement
Charge Modes:
simple: The entitlement values in the customer configuration are used only for reporting and do not impact charge calculations (a customer with 100 GB of ingest and a 50 GB entitlement is charged for 100 GB). This was the only option prior to dvvy 1.5.0.
over_entitlement: The entitlement values in the customer configuration change how charges are calculation (a customer with 100 GB of ingest and a 50 GB entitlement is charged for 50 GB).
-
currencyUnit: The currency symbol to use in dashboards. Possible values: $, €,£, etc.
-
currencyUnitPosition: The position of the currency symbol in dashboards. Possible values: before, after
-
dvvyAdmin: A comma-separated list of Splunk usernames that have access to all dvvy customer data in dashboards.
-
indexerTargetGB: The daily indexing target per indexer in GB based on your architecture and sizing. Used for Indexer Load (IDXL) calculation, a simple methodology to represent indexer consumption as a whole.
-
kvkitBaseURL: The URL of the kvkit application if installed.
-
splunkAdminCost: The daily cost of a Splunk Administrator/Engineer to use in admin chargeback calculations (ex: $150,000 yearly salary = $410.95 daily)
-
splunkAdminFlat: A flat daily fee to apply to all customers as part of admin chargeback.
-
splunkAdminForwarder: The quantity of forwarders a single Splunk admin can support (used to calculate Splunk Admin Load)
-
splunkAdminLicenseGB: The amount of license (GB) a single Splunk admin can support (used to calculate Splunk Admin Load)
-
splunkAdminStorage: The amount of storage (TB) a single admin can support (used to calculate Splunk Admin Load)
-
splunkAdminUsers: The number of users a single admin can support (used to calculate Splunk Admin Load)
See Recommended Configuration Workflow below for recommendations of working with data in KV store collections.
Customers are defined in the dvvyGroups KV store collection.
Below are the available parameters:
-
costForwarder: The daily rate per 100 forwarders
-
costIndexer: The daily rate for indexer load
-
costLicenseGB: The daily rate for 1 GB of data ingest
-
costRTSearchTime: The daily rate for 1 second of real-time search runtime
-
costSearchTime: The daily rate for 1 second of search runtime
-
costStorageColdTB: The daily rate for 1 TB of cold storage
-
costStorageHotWarmTB: The daily rate for 1 TB of hot/warm storage
-
costVCPU: The daily rate for 1 VCPU
-
group: The org_id value for the customer (i.e., "noc" or "soc" from the earlier example)
-
groupDisplay: The customer display name (i.e., friendly name) for dashboard presentation
-
licenseEntitlementGB: The amount of data ingest (GB/day) allocated to the customer for chargeback and reporting purposes
-
rtSearchTimeEntitlement: The amount of real-time search runtime (seconds) allocated to the customer for chargeback and reporting purposes
-
searchTimeEntitlment: The amount of search runtime (seconds) allocated to the customer for chargeback and reporting purposes
-
splunkRole: A comma-separated list of Splunk roles associated with the customer (used for Splunk Admin Load charge calculation)
-
storageEntitlementGB: The total amount of storage allocated to a given group for reporting purposes
-
techContact: A comma-separated list of Splunk usernames that are affiliated with the group and can access dvvy
-
vcpuEntitlement: The quantity of virtual CPUs (vCPU) allocated to the customer for chargeback and reporting purposes
See Recommended Configuration Workflow below for recommendations of working with data in KV store collections.
Update the CUSTOMER_IDENTIFIER macro with the field you've defined for your Org ID. By default, the value is org_id. It can be set to anything. Once data is flowing with your Org ID defined and the other app configuration is in place, run the KV - Track Data report. This will write all data to the dvvyData collection. This report is scheduled to run daily out of the box. Running it manually is only required to accelerate the implementation process.
Do not manually update dvvyData contents. If you need to add a costCenter or make modifications, it is recommended that you alter the logic of the KV - Track Data report.
The KV - Track Data report will add the following information to the dvvyData collection for any untracked data source:
-
group: The Org ID associated with the source type
-
idx: The index
-
st: The source type
-
costCenter: The cost center will be set to "default" unless search is modified with lookup
-
timestamp: Timestamp of when the data source was added to dvvy (added automatically)
Run the KV - Splunk Infrastructure report to capture system specs of indexers and search heads in your environment. This search is not scheduled by default - please schedule to effectively capture any changes in your environment at a frequency that makes sense.
If your environment changes (e.g., add/remove an indexer or search head) and you do not run the report, VCPU reporting will not be accurate.
The dvvy app relies on three (3) summary indexes for operation. Create the following indexes on your indexers:
-
dvvy_event_summary: Customer event counts from the Utilization Summary - Events search.
-
dvvy_usage_summary: Customer utilization from all other Utilization Summary searches.
-
dvvy_charge_summary: Customer charge calculations from all Cost Summary searches.
Optionally add accounting cost centers to the dvvyCostCenters collection to further partition the data for reporting.
-
costCenter: The cost center code
-
costCenterDescription: The cost center display name for dashboards
Once added to dvvyCostCenters, you will need to add costCenter to each data source in dvvyData. This can be done manually or programmatically. Consider creating a lookup table to populate this field.
If cost centers aren't required, you might this field as an internal identifier or reference to enhance reporting.
When you install dvvy, all of the necessary utilization and calculation reports are enabled and scheduled. By default, all jobs are scheduled to run overnight for the prior day's events with the exception of the "Utilization Summary - License GB" search that runs every five (5) minutes by default.
Saved searches in Splunk run as the owner by default. When dvvy is installed, ownership will not be set ('nobody' ownership in meta) . This will cause the saved searches to interpret the report schedules as UTC and then apply the timezone offset (i.e., -4 for EDT, etc.). This may break the schedule by causing the searches to span two days. All reports must execute during the same day.
Set ownership on all saved searches to an admin's account with the correct local time zone via Reassign Knowledge Objects and restart Splunk. After restart, confirm that the saved searches starting with Utilization Summary - Events is intended to kick off at 12:03 AM nightly.
Although the schedule, time ranges/frequency, etc. can be changed to meet your requirements, daily search jobs must execute in a specific order due to dependencies. Consider removing the | collect command when testing searches to avoid writing data to summaries prematurely.
| Report | Schedule Sequence | Description | Dependencies | Relevant Configuration | Index |
|---|---|---|---|---|---|
| Utilization Summary - Events | 1 | Percentage of data ownership by customer based on index and sourcetype. This allows for overlapping data sources (i.e., same index/sourcetype used by multiple customers). | dvvy_event_summary | ||
| Utilization Summary - License GB | 2 | Data ingest by customer. |
|
dvvy_usage_summary | |
| Utilization Summary - Storage | 3 | Hot/warm and cold storage utilization by customer. |
|
dvvy_usage_summary | |
| Utilization Summary - Forwarders | 4 | Quantity of forwarders by customer. | dvvy_usage_summary | ||
| Utilization Summary - Search | 5 | Runtime (seconds) for search activity by customer. | dvvy_usage_summary | ||
| Utilization Summary - CPU | 6 | Average vCPU utilization and related metrics by customer. | dvvy_usage_summary | ||
| Cost Summary - License GB | 7 | License chargeback by customer. |
|
|
dvvy_charge_summary |
| Cost Summary - Storage | 8 | Storage chargeback by customer. |
|
|
dvvy_charge_summary |
| Cost Summary - Search | 9 | Search runtime (seconds) chargeback by customer. |
|
|
dvvy_charge_summary |
| Cost Summary - RT Search | 10 | Realtime search runtime (seconds) chargeback by customer. |
|
|
dvvy_charge_summary |
| Cost Summary - CPU | 11 | vCPU chargeback by customer that compares search, realtime search, and data ingest between customers against available resources. |
|
|
dvvy_charge_summary |
| Cost Summary - Forwarders | 12 | Forwarder chargeback by customer (per 100 forwarders). |
|
dvvy_charge_summary | |
| Cost Summary - Indexer Simple | 13 | Indexer chargeback based on target sizing. Chargeback is based on Indexer Load (IDXL), where 1 IDXL = 1 indexer based on sizing. |
|
|
dvvy_charge_summary |
| Utilization Summary - Admin - License GB | 14 | Splunk Admin Load (SAL) based on per-FTE license quantity |
|
dvvy_usage_summary | |
| Utilization Summary - Admin - Storage | 15 | Splunk Admin Load (SAL) based on per-FTE storage quantity. |
|
dvvy_usage_summary | |
| Utilization Summary - Admin - Users | 16 | Splunk Admin Load (SAL) based on per-FTE user quantity. |
|
dvvy_usage_summary | |
| Utilization Summary - Admin - Forwarders | 17 | Splunk Admin Load (SAL) based on per-FTE forwarder quantity. |
|
dvvy_usage_summary | |
| Cost Summary - Flat Admin Fee | 18 | Flat admin fee applied to every customer. |
|
dvvy_charge_summary | |
| Cost Summary - Admin | 19 | Combined Splunk Admin Load (SAL) chargeback for license, storage, users, and forwarders. |
|
dvvy_charge_summary |
Contact us for scripts and other assets to streamline dvvy testing before going live.
If you run the searches in the sequence above (manually or via our handy shell script), dashboards should populate. All dashboards only search against the *_cost_summary indexes. Dashboard filters are restricted to customer affiliation as defined by techContact in dvvyGroups. Any users listed in dvvyAdmin in the dvvyConfig collection will have access to all app data in the dashboards. Once everything is populated to your liking, to include the charge calculations, clean (or | delete) all of dvvy indexes so that reporting is not skewed.
All RedFactor customers with a dvvy app license automagically have a zero-cost kvkit license. Kvkit is a lightweight Node/Express application that sits outside of Splunk and interacts with the the KV store via REST API. It can be used to create web-based forms that can be used to easily update dvvy configuration.
In lieu of kvkit, please us the Lookup File Editor app.
The dvvy app leverages KV store collections for all request operations. The table below lists the collections and their role.
| Collection Name | Description |
|---|---|
| dvvyConfig | Global app configuration options |
| dvvyCostCenters | Cost center information |
| dvvyData | Data sources tracked by dvvy |
| dvvyGroups | Group ("customer") configuration |
| dvvySplunkInfrastructure | Splunk environment information used for vCPU chargeback |
Since some of the app data is stored in KV store collections and collections are susceptible to accidental deletion or overwrite (e.g., unintentional outputlookup by an admin), it is highly recommended that you regularly backup all dvvy collections to prevent data loss.
If you are using the kvkit application, you can easily make on-demand backups or schedule automated backups based on crontab syntax.
Backups can be performed via CLI, like so:
$SPLUNK_HOME/bin/splunk backup kvstore [-archiveName <archive>] [-collectionName <collection>] [-appName <app>]
Using the above syntax as a guide, running this command:
/opt/splunk/bin/splunk backup kvstore -archiveName dvvyData -collectionName dvvyData -appName dvvy
will generate a .tar.gz archive in /opt/splunk/var/lib/splunk/kvstorebackup.
The archive will contain a JSON file with the contents of the dvvyData collection which could be used to restore.
Consider scheduling this command with cron (or equivalent) or creating a simple shell script to streamline the process of backing up each dvvy collection. Please see Backup and Restore the KV store in the Admin Manual for official Splunk guidance on the topic.