Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
---
title: Workshop Overview
linkTitle: 1. Workshop Overview
weight: 1
archetype: chapter
time: 2 minutes
description: Workshop Overview
---

**Introduction**
The goal of this workshop is to give you hands-on experience troubleshooting an issue using Splunk Observability Cloud to identify its root cause. We’ve provided a fully instrumented microservices-based application that actually mimic a wire transfer workflow that is running on Kubernetes, which sends metrics, traces, and logs to Splunk Observability Cloud for real-time analysis.

**Who Should Attend?**
This workshop is ideal for anyone looking to gain practical knowledge of Splunk Observability. It's designed for individuals with little or no prior experience with the platform.

**What You’ll Need**
All you need is your laptop and a browser with access to external websites. The workshop can be attended either in-person or via Zoom. If you don’t have the Zoom client installed, you can still join using your browser.

**Workshop Overview**
In this 3-hour session, we’ll cover the fundamentals of Splunk Observability—the only platform offering streaming analytics and NoSample Full Fidelity distributed tracing—in an interactive, hands-on setting. Here's what you can expect:

- **OpenTelemetry**
Learn why OpenTelemetry is essential for modern observability and how it enhances visibility into your systems.

- **Tour of the Splunk Observability User Interface**
Take a guided tour of Splunk Observability Cloud’s interface, where we’ll show you how to navigate the five key components: APM, Log Observer, and Infrastructure.

- **Splunk Application Performance Monitoring (APM)**
Gain end-to-end visibility of your customers' request path using APM traces. You’ll explore how telemetry from various services is captured and visualized in Splunk Observability Cloud, helping you detect anomalies and errors.

- **Splunk Log Observer (LO)**
Learn how to leverage the "Related Content" feature to easily navigate between components. In this case, we’ll move from an APM trace to the related logs for deeper insight into issues.

By the end of this session, you'll have gained practical experience with Splunk Observability Cloud and a solid understanding of how to troubleshoot and resolve issues across your application stack.
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
---
title: What is OpenTelemetry & why should you care?
linkTitle: 2. OpenTelemetry
weight: 2
archetype: chapter
time: 2 minutes
description: Learn about OpenTelemetry and why you should care about it.
---

## OpenTelemetry

With the rise of cloud computing, microservices architectures, and ever-more complex business requirements, the need for Observability has never been greater. Observability is the ability to understand the internal state of a system by examining its outputs. In the context of software, this means being able to understand the internal state of a system by examining its telemetry data, which includes **metrics**, **traces**, and **logs**.

To make a system observable, it must be instrumented. That is, the code must emit traces, metrics, and logs. The instrumented data must then be sent to an Observability back-end such as **Splunk Observability Cloud**.

| Metrics | Traces | Logs |
|:-------:|:------:|:----:|
| _**Do I have a problem?**_ | _**Where is the problem?**_ | _**What is the problem?**_ |

OpenTelemetry does two important things:

* Allows you to **own** the data that you generate rather than be stuck with a proprietary data format or tool.
* Allows you to learn **a single set** of APIs and conventions

These two things combined enable teams and organizations the flexibility they need in today’s modern computing world.

There are a lot of variables to consider when getting started with Observability, including the all-important question: _"How do I get my data into an Observability tool?"_. The industry-wide adoption of OpenTelemetry makes this question easier to answer than ever.

## Why Should You Care?

OpenTelemetry is completely open-source and free to use. In the past, monitoring and Observability tools relied heavily on proprietary agents meaning that the effort required to change or set up additional tooling required a large amount of changes across systems, from the infrastructure level to the application level.

Since OpenTelemetry is vendor-neutral and supported by many industry leaders in the Observability space, adopters can switch between supported Observability tools at any time with minor changes to their instrumentation. This is true regardless of which distribution of OpenTelemetry is used – like with Linux, the various distributions bundle settings and add-ons but are all fundamentally based on the community-driven OpenTelemetry project.

Splunk has fully committed to OpenTelemetry so that our customers can collect and use **ALL** their data, in any type, any structure, from any source, on any scale, and all in real-time. OpenTelemetry is fundamentally changing the monitoring landscape, enabling IT and DevOps teams to bring data to every question and every action. You will experience this during these workshops.

![OpenTelemetry Logo](images/otel.png)
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
---
title: Home Page
linkTitle: 1.1 Home Page
weight: 2
time: 5 minutes
---

After you have registered and logged into Splunk Observability Cloud you will be taken to the home or landing page. Here, you will find several useful features to help you get started.

![home page](../images/home-screen.png)

1. **Explore your data pane:** Displays which integrations are enabled and allows you to add additional integrations if you are an Administrator.
2. **Documentation pane:** Training videos and links to documentation to get you started with Splunk Observability Cloud.
3. **Recents pane:** Recently created/visited dashboards and/or detectors for quick access.
4. **Main Menu pane:** Navigate the components of Splunk Observability Cloud.
5. **Org Switcher:** Easily switch between Organizations (if you are a member of more than one Organization).
6. **Expand/Contract Main Menu:** Expand **>>** / Collapse **<<** the main menu if space is at a premium.

Let's start with our first exercise:

{{% notice title="Exercise" style="green" icon="running" %}}

* Expand the Main Menu and click on **Settings**.
* Check in the **Org Switcher** if you have access to more than one Organization.

{{% /notice %}}

{{% notice title="Tip" style="primary" icon="lightbulb" %}}
If you have used Splunk Observability before, you may be placed in an Organization you have used previously. Make sure you are in the correct workshop organization. Verify this with your instructor if you have access to multiple Organizations.
{{% /notice %}}

{{% notice title="Exercise" style="green" icon="running" %}}

* Click **Onboarding Guidance** (Here you can toggle the visibility of the onboarding panes. This is useful if you know the product well enough, and can use the space to show more information).
* Hide the Onboarding Content for the **Home Page**.
* At the bottom of the menu, select your preferred appearance: **Light**, **Dark** or **Auto** mode.
* Did you also notice this is where the **Sign Out** option is? Please don't 😊 !
* Click **<** to get back to the main menu.

{{% /notice %}}

Next, let's check out **Splunk Real User Monitoring (RUM)**.
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
---
title: Log on FAQ
weight: 99
hidden: true
---

This FAQ will address some of the more common issues we have encountered when logging into the Workshop.

### 1. Invite email or password renewal email not arriving

The first step to take is to search for an email from **<noreply@signalfx.com>**, across all your email folders, as this is the address used to send the invite and password renewal emails. If you don't see the email, check your spam/junk folder.

If you are sure the email does not exist your email, ask the Instructor to verify the email used for the workshop and have him/her resend the invite.

If this fails, another solution is to provide the Instructor with a different email address (private e-mail address for example) and have him/her resend the invite.

---

### 2. Password not accepted

The requirements for a password in Splunk Observability Cloud are:

* **Must** be between 8 and 32 characters
* **Must** contain at least one capital letter
* **Must** have at least one number
* **Must** have at least one symbol (e.g. !@#$%^&*()_+)

---

### 3. Invalid or unknown password

The system does not recognize the password and username combination, please click on the reset password link to try and reset your password.
You will be asked to provide a password. If that account exists, an email will be sent to allow you to reset your password. follow the instructions in that email.

If no email arrives or your username is not recognized, reach out to your instructor for assistance.

---

### 4. Other options

To Be Completed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
---
title: Getting Started
linkTitle: 1. Getting Started
weight: 1
time: 2 minutes
description: Learn how to get started with Splunk Observability Cloud.
---

### 1. Sign in to Splunk Observability Cloud

You should have received an e-mail from Splunk inviting you to the Workshop Org. This e-mail will look like the screenshot below, if you cannot find it, please check your Spam/Junk folders or inform your Instructor. You can also check for other solutions in our [**login F.A.Q.**](99-login-faq).

To proceed click the **Join Now** button or click on the link provided in the e-mail.

If you have already completed the registration process you can skip the rest and proceed directly to Splunk Observability Cloud and log in:

* [**https://app.eu0.signalfx.com (EMEA)**](https://app.eu0.signalfx.com)
* [**https://app.us1.signalfx.com (APAC/AMER)**](https://app.us1.signalfx.com)

![email](images/invite-email.png?width=25vw)

If this is your first time using Splunk Observability Cloud, you will be presented with the registration form. Enter your full name, and desired password. Please note that the password requirements are:

* **Must** be between 8 and 32 characters
* **Must** contain at least one capital letter
* **Must** have at least one number
* **Must** have at least one symbol (e.g. !@#$%^&*()_+)

Click the checkbox to agree to the terms and conditions and click the **SIGN IN NOW** button.

![User-Setup](images/enter-password.png?width=25vw)
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
title: Application Performance Monitoring Home page
linkTitle: 2.1 APM Home Page
weight: 2
---

Click **APM** in the main menu, the APM Home Page is made up of 3 distinct sections:

![APM page](../images/apm-main.png)

1. **Onboarding Pane Pane:** Training videos and links to documentation to get you started with Splunk APM.
2. **APM Overview Pane:** Real-time metrics for the Top Services and Top Business Workflows.
3. **Functions Pane:** Links for deeper analysis of your services, tags, traces, database query performance and code profiling.

The **APM Overview** pan provides a high-level view of the health of your application. It includes a summary of the services, latency and errors in your application. It also includes a list of the top services by error rate and the top business workflows by error rate (a business workflow is the start-to-finish journey of the collection of traces associated with a given activity or transaction and enables monitoring of end-to-end KPIs and identifying root causes and bottlenecks).

{{% notice title=" About Environments" style="info" %}}

To easily differentiate between multiple applications, Splunk uses **environments**. The naming convention for workshop environments is **[NAME OF WORKSHOP]-workshop**. Your instructor will provide you with the correct one to select.

{{% /notice %}}

{{% notice title="Exercise" style="green" icon="running" %}}

* Verify that the time window we are working with is set to the last 15 minutes (**-15m**).
* Change the environment to the workshop one by selecting its name from the drop-down box and make sure that is the only one selected.
{{< tabs >}}
{{% tab title="Question" %}}
**What can you conclude from the *Top Services by Error Rate* chart?**
{{% /tab %}}
{{% tab title="Answer" %}}
**The *wire-transfer-service* has a high error rate**
{{% /tab %}}
{{< /tabs >}}
<!--
* Click on the Explore Tile in the Function Pane. This will bring us to the automatically generated map of our services. This map shows how the services interact together based on the trace data being sent to Splunk Observability Cloud.
-->
{{% /notice %}}

If you scroll down the Overview Page you will notice some services listed have **Inferred Service** next to them.

Splunk APM can infer the presence of the remote service, or inferred service if the span calling the remote service has the necessary information. Examples of possible inferred services include databases, HTTP endpoints, and message queues. Inferred services are not instrumented, but they are displayed on the service map and the service list.

Next, let's check out **Splunk Log Observer (LO)**.
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
title: Application Performance Monitoring Overview
linkTitle: 2. APM Overview
weight: 3
time: 5 minutes
---

Splunk APM provides a **NoSample** end-to-end visibility of every service and its dependency to solve problems quicker across monoliths and microservices. Teams can immediately detect problems from new deployments, confidently troubleshoot by scoping and isolating the source of an issue, and optimize service performance by understanding how back-end services impact end users and business workflows.

**Real-time monitoring and alerting:** Splunk provides out-of-the-box service dashboards and automatically detects and alerts on RED metrics (rate, error and duration) when there is a sudden change.

**Dynamic telemetry maps:** Easily visualize service performance in modern production environments in real-time. End-to-end visibility of service performance from infrastructure, applications, end users, and all dependencies helps quickly scope new issues and troubleshoot more effectively.

**Intelligent tagging and analysis:** View all tags from your business, infrastructure and applications in one place to easily compare new trends in latency or errors to their specific tag values.

**AI-directed troubleshooting identifies the most impactful issues:** Instead of manually digging through individual dashboards, isolate problems more efficiently. Automatically identify anomalies and the sources of errors that impact services and customers the most.

**Complete distributed tracing analyses every transaction:** Identify problems in your cloud-native environment more effectively. Splunk distributed tracing visualizes and correlates every transaction from the back-end and front-end in context with your infrastructure, business workflows and applications.

**Full stack correlation:** Within Splunk Observability, APM links traces, metrics, logs and profiling together to easily understand the performance of every component and its dependency across your stack.

**Monitor database query performance:** Easily identify how slow and high execution queries from SQL and NoSQL databases impact your services, endpoints and business workflows — no instrumentation required.

![Architecture Overview](./images/arch-overview.png)
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
---
title: Log Observer Home Page
linkTitle: 3.1 Log Observer Home Page
weight: 2
---

Click **Log Observer** in the main menu, the Log Observer Home Page is made up of 4 distinct sections:

![Lo Page](../images/log-observer-main.png)

1. **Onboarding Pane:** Training videos and links to documentation to get you started with Splunk Log Observer.
2. **Filter Bar:** Filter on time, indexes, and fields and also Save Queries.
3. **Logs Table Pane:** List of log entries that match the current filter criteria.
4. **Fields Pane:** List of fields available in the currently selected index.

{{% notice title=" Splunk indexes" style="info" %}}

Generally, in Splunk, an "index" refers to a designated place where your data is stored. It's like a folder or container for your data. Data within a Splunk index is organized and structured in a way that makes it easy to search and analyze. Different indexes can be created to store specific types of data. For example, you might have one index for web server logs, another for application logs, and so on.

{{% /notice %}}

{{% notice title="Tip" style="primary" icon="lightbulb" %}}

If you have used Splunk Enterprise or Splunk Cloud before, you are probably used to starting investigations with logs. As you will see in the following exercise, you can do that with Splunk Observability Cloud as well. This workshop, however, will use all the **OpenTelemetry** signals for investigations.

{{% /notice %}}

Let's run a little search exercise:

{{% notice title="Exercise" style="green" icon="running" %}}

* Set the time frame to **-15m**.
* Click on {{% button style="gray" %}}Add Filter{{% /button %}} in the filter bar then click on **Fields** in the dialog.
* Type in **cardType** and select it.
* Under **Top values** click on **visa**, then click on **=** to add it to the filter.
* Click {{% button style="blue" %}}Run search{{% /button %}}

![logo search](../images/log-filter-bar.png?width=920px)

* Click on one of the log entries in the Logs table to validate that the entry contains `cardType: "visa"`.
* Let's find all the wire transfer orders that have been compelted. Click on {{% button style="gray" %}}Clear All{{% /button %}} in the filter bar to remove the previous filter.
* Click again on {{% button style="gray" %}}Add Filter{{% /button %}} in the filter bar, then select **Keyword**. Next just type `order` in the **Enter Keyword...** box and press enter.
* Click {{% button style="blue" %}}Run search{{% /button %}}
* You should now only have log lines that contain the word `order`. There are still a lot of log lines -- some of which may not be our service -- so let's filter some more.
* Add another filter, this time select the **Fields** box, then type `severity` in the **Find a field ...** search box and select it.
![severity](../images/find-severity.png?width=15vw&classes=left)
* Under **Top values** click on **error**, then click on **=** to add it to the filter.
* Click {{% button style="blue" %}}Run search{{% /button %}}
* You should now have a list of wire transfer orders that failed to complete for the last 15 minutes.

{{% /notice %}}

Next, let's check out **Splunk Synthetics**.
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
title: Log Observer Overview
linkTitle: 3. Log Observer Overview
weight: 4
time: 5 minutes
---

Log Observer Connect allows you to seamlessly bring in the same log data from your Splunk Platform into an intuitive and **no-code** interface designed to help you find and fix problems quickly. You can easily perform log-based analysis and seamlessly correlate your logs with Splunk Infrastructure Monitoring’s real-time metrics and Splunk APM traces in one place.

**End-to-end visibility:** By combining the powerful logging capabilities of Splunk Platform with Splunk Observability Cloud’s traces and real-time metrics for deeper insights and more context of your hybrid environment.

**Perform quick and easy log-based investigations:** By reusing logs that are already ingested in Splunk Cloud Platform or Enterprise in a simplified and intuitive interface (no need to know SPL!) with customizable and out-of-the-box dashboards

**Achieve higher economies of scale and operational efficiency:** By centralizing log management across teams, breaking down data and team silos, and getting better overall support

![Logo graph](./images/logo-image-loop.png)
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading