-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #13694 from newrelic/revert-13691-revert-13613-Loo…
…kup-tables---new-doc New docs on lookup tables
- Loading branch information
Showing
11 changed files
with
267 additions
and
24 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,62 @@ | ||
--- | ||
title: "Upload CSV-format lookup tables" | ||
metaDescription: 'In the New Relic logs UI, upload CSV-format lookup tables and use that data in combination with other New Relic data.' | ||
--- | ||
|
||
import logsLookupTableUi from 'images/logs_screenshot-crop_lookup-table-ui.webp' | ||
|
||
Our **lookups** feature lets you enrich your log data, and other New Relic-stored data, with data about your business that you define in a CSV file. | ||
|
||
## Why use lookups? [#overview] | ||
|
||
When you upload a lookup table, you can then use that data to enrich your queries of New Relic data. For example, you might upload a table that maps host IDs to human-readable host names, and then use that to create a chart that displays the human-readable host names. | ||
|
||
Lookup tables help you: | ||
|
||
* Query data that isn't present in your New Relic account | ||
* Make your telemetry data easier to understand | ||
* Group data in custom ways | ||
|
||
For examples of queries using lookups, see [How to query lookup table data](/docs/query-your-data/nrql-new-relic-query-language/nrql-query-tutorials/lookups). | ||
|
||
## Add and manage tables [#add-table] | ||
|
||
<img | ||
title="Upload a csv lookup table" | ||
alt="Upload a csv lookup table" | ||
src={logsLookupTableUi} | ||
/> | ||
|
||
The UI for uploading lookup tables is located in the logs UI, but you can use your lookup table data when querying any data type, not just logs. | ||
|
||
To find the lookup table UI: From [one.newrelic.com](https://one.newrelic.com), click **Logs**, and then click **Lookup tables**. From there, you can upload, download, update, and delete tables. | ||
|
||
Some tips when adding a table: | ||
|
||
* The table name is what you'll use to reference that table when you write a [NRQL query](/docs/query-your-data/nrql-new-relic-query-language/nrql-query-tutorials/lookups). | ||
* Table data isn't obfuscated, so avoid uploading sensitive information. | ||
* It can take several minutes for an added or updated table to be available in a NRQL query. | ||
|
||
When you delete a table, that data is no longer available in NRQL queries, so you should try to make sure a deleted table's data isn't being used in important dashboards. | ||
|
||
[Learn more table format requirements and other details](#requirements). | ||
|
||
## Query your data [#query] | ||
|
||
If you're ready to use your table in a query, see [How to query lookup table data](/docs/query-your-data/nrql-new-relic-query-language/nrql-query-tutorials/lookups). | ||
|
||
## Requirements and technical details [#requirements] | ||
|
||
Some more details about lookup tables: | ||
|
||
* Tables can only be queried from the [account](/docs/accounts/accounts-billing/account-structure/new-relic-account-structure/#organization-accounts) in which the table was uploaded. | ||
* Each [account](/docs/accounts/accounts-billing/account-structure/new-relic-account-structure/#organization-accounts) has a limit of 20 lookup tables. | ||
* Lookup table data can't be used in NRQL alert conditions. | ||
|
||
In addition to [the general requirements for a CSV file](https://www.rfc-editor.org/rfc/rfc4180), here are our requirements for uploading lookup table files: | ||
|
||
* Each row must have the same number of columns as the header and there must be at least two columns. | ||
* Table names must conform to [the rules for event names](/docs/data-apis/ingest-apis/event-api/introduction-event-api/#limits) | ||
* Do not use [reserved words](/docs/data-apis/custom-data/custom-events/data-requirements-limits-custom-event-data/#reserved-words) for table names or column header values. | ||
* Max file size: 4 MB | ||
* Max of 20,000 rows |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
160 changes: 160 additions & 0 deletions
160
.../query-your-data/nrql-new-relic-query-language/nrql-query-tutorials/lookups.mdx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,160 @@ | ||
--- | ||
title: How to query lookup table data | ||
tags: | ||
- Query your data | ||
- 'NRQL: New Relic Query Language' | ||
- NRQL query tutorials | ||
metaDescription: In New Relic, how to query data added via CSV lookup tables. | ||
redirects: | ||
--- | ||
|
||
import nrqlExampleStatusCodes from 'images/nrql_screenshot-crop_example-status-codes.webp' | ||
|
||
import nrqlLookupQueryTranslateItemIds from 'images/nrql_screenshot-crop_lookup-query-translate-item-ids.webp' | ||
|
||
import nrqlLookupQueryGeoip from 'images/nrql_screenshot-crop_lookup-query-geoip.webp' | ||
|
||
When you [upload CSV-format lookup tables](/docs/logs/ui-data/lookup-tables-ui), you can use the `lookup()` function to access that data in your NRQL queries. | ||
|
||
## Why use lookup tables? [#why] | ||
|
||
For why you'd use lookup tables and how to upload them, see [Lookup tables](/docs/logs/ui-data/lookup-tables-ui). | ||
|
||
## Basic query syntax [#basic-syntax] | ||
|
||
Let's say you've named your table `storeNames`. This query will select all data from that table: | ||
|
||
```sql | ||
FROM lookup(storeNames) | ||
SELECT * | ||
``` | ||
|
||
This query will select some specific attributes from that same table: | ||
|
||
```sql | ||
FROM lookup(storeNames) | ||
SELECT store_ID, store_name, description | ||
``` | ||
|
||
## Query examples [#query-with-data] | ||
|
||
The primary benefit of lookup tables is that you can use queries that combine that data with your New Relic-stored telemetry data. | ||
|
||
Here are some query examples: | ||
|
||
<CollapserGroup> | ||
<Collapser | ||
id="avoid-hardcording" | ||
title="Avoid hardcoding a long list of hosts" | ||
> | ||
|
||
This query avoids hardcoding a long list of hosts by querying host names contained in a lookup table: | ||
|
||
```sql | ||
FROM Log | ||
SELECT count(*) | ||
WHERE hostname IN (FROM lookup(myHosts) SELECT uniques(myHost)) | ||
``` | ||
</Collapser> | ||
|
||
<Collapser | ||
id="using-join" | ||
title="Query using JOIN" | ||
> | ||
|
||
Using `JOIN` queries can make your data more understandable. For example, this query for a custom event type uses the `storeNames` table to show the store names along with the total sales. Also notice this allows a limit of 10,000: that's because lookup tables support a higher limit than other NRQL data types. | ||
|
||
```sql | ||
FROM StoreEvent | ||
JOIN (FROM lookup(storeNames) SELECT store_ID as storeId, storeName AS name LIMIT 10000) ON shopId = storeId | ||
SELECT shopId, storeName, totalSales | ||
``` | ||
</Collapser> | ||
<Collapser | ||
id="status-codes" | ||
title="Translate status codes" | ||
> | ||
Here's a query that translates status codes to readable summaries of the status: | ||
|
||
```sql | ||
FROM Transaction | ||
JOIN (FROM lookup(statusCodeTable) SELECT status_code, status_summary, status_definition) | ||
ON http.statusCode=status_code | ||
SELECT count(*) FACET status_summary | ||
``` | ||
|
||
Here are some example results: | ||
|
||
<img | ||
title="Screenshot of query for lookups translating status codes" | ||
alt="Screenshot of query for lookups translating status codes" | ||
src={nrqlExampleStatusCodes} | ||
/> | ||
</Collapser> | ||
|
||
<Collapser | ||
id="item-ids" | ||
title="Translate item IDs" | ||
> | ||
|
||
This query shows how several NRQL features can work together to get business information from queries of log data. The query extracts information about items from log messages, and uses `JOIN` to get user-friendly item names and to generate a table of item names and the number of items stored. | ||
|
||
```sql | ||
WITH aparse(message, 'POST to carts: * body: {"itemId":"*","unitPrice":*}%') AS (URL, ItemID, Price) | ||
FROM Log | ||
JOIN (FROM lookup(itemNames) SELECT ItemID, itemName) ON ItemID | ||
SELECT count(*) FACET itemName | ||
WHERE message LIKE 'POST to carts%' | ||
SINCE 30 days ago | ||
``` | ||
|
||
Here are some example results: | ||
|
||
<img | ||
title="Screenshot of query for lookups translating item IDs" | ||
alt="Screenshot of query for lookups translating item IDs" | ||
src={nrqlLookupQueryTranslateItemIds} | ||
/> | ||
|
||
</Collapser> | ||
|
||
<Collapser | ||
id="geoip" | ||
title="Use geographic info to analyze locations" | ||
> | ||
|
||
This query combines lookups and [GeoIP](/docs/logs/ui-data/parsing#geo) to find which locations have the most unsuccessful statuses: | ||
|
||
```sql | ||
FROM Log | ||
JOIN (FROM lookup(statusCodeTable) SELECT status_code, status_summary, status_definition) | ||
ON CacheResponseStatus=status_code | ||
SELECT count(*) WHERE ClientIP.countryName IS NOT NULL AND status_summary != 'Success' | ||
FACET ClientIP.countryName, status_summary, CacheResponseStatus | ||
SINCE 1 day ago LIMIT MAX | ||
``` | ||
|
||
Here are some example results: | ||
|
||
<img | ||
title="Screenshot of query for GeoIP info" | ||
alt="Screenshot of query for GeoIP info" | ||
src={nrqlLookupQueryGeoip} | ||
/> | ||
|
||
</Collapser> | ||
</CollapserGroup> | ||
|
||
## Technical details about querying [#limitations] | ||
|
||
The maximum number of responses returned when using `LIMIT` is 20,000. | ||
|
||
You can't use lookup data for NRQL alert conditions. | ||
|
||
The following NRQL functions are not supported with lookup queries: | ||
|
||
* `TIMESERIES` | ||
* `COMPARE WITH` | ||
* `EXTRAPOLATE` | ||
|
||
But these clauses can be used if the lookup query is contained in an inner query. For an example of this, see [this query](#item-ids). |
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters