Skip to content

feat: more approachable & in-depth indexer intro #2055

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ElliotFriend merged 1 commit into from
Dec 19, 2025
Merged

feat: more approachable & in-depth indexer intro #2055

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
143 changes: 3 additions & 140 deletions config/sidebars.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,43 +21,10 @@ const sidebars: SidebarsConfig = {
dirName: 'tokens',
},
],
data_overview: [
'data/README',
data: [
{
type: 'category',
label: 'Analytics',
link: { type: 'doc', id: 'data/analytics/README' },
items: [
{ type: 'ref', id: 'data/analytics/hubble/README', label: 'Hubble' },
{ type: 'ref', id: 'data/analytics/analytics-providers/analytics-providers', label: 'Data Analytics Providers' },
]
},
{
type: 'category',
label: 'APIs',
link: { type: 'doc', id: 'data/apis/README' },
items: [
{ type: 'ref', id: 'data/apis/rpc/README', label: 'RPC' },
{ type: 'ref', id: 'data/apis/horizon/README', label: 'Horizon' },
{ type: 'ref', id: 'data/apis/migrate-from-horizon-to-rpc', label: 'Migrate Horizon to RPC' },
]
},
{
type: 'category',
label: 'Indexers',
link: { type: 'doc', id: 'data/indexers/README' },
items: [
{ type: 'ref', id: 'data/indexers/build-your-own/README', label: 'Build Your Own Indexer' },
{ type: 'ref', id: 'data/indexers/indexer-providers/indexer-providers', label: 'Indexer Providers' },
]
},
{
type: 'category',
label: 'Oracles',
link: { type: 'doc', id: 'data/oracles/README' },
items: [
{ type: 'ref', id: 'data/oracles/oracle-providers', label: 'Oracle Providers' },
]
type: 'autogenerated',
dirName: 'data',
},
],
tools: [
Expand All @@ -81,110 +48,6 @@ const sidebars: SidebarsConfig = {
dirName: 'validators',
},
],
hubble: [
{
type: 'category',
label: 'Hubble',
items: [
{
type: "autogenerated",
dirName: "data/analytics/hubble",
},
],
collapsible: false,
},
],
galexie: [
{
type: 'category',
label: 'Galexie',
items: [
{
type: "autogenerated",
dirName: "data/indexers/build-your-own/galexie",
},
],
collapsible: false,
},
],
ingest: [
{
type: 'category',
label: 'Ingest SDK',
items: [
{
type: "autogenerated",
dirName: "data/indexers/build-your-own/ingest-sdk",
},
],
collapsible: false,
},
],
soroban_rpc: [
{
type: "category",
label: "Stellar RPC",
items: [
{
type: 'autogenerated',
dirName: 'data/apis/rpc',
},
],
collapsible: false,
},
],
analytics: [
{
type: 'category',
label: 'Analytics',
items: [
{
type: "autogenerated",
dirName: "data/analytics",
},
],
collapsible: false,
},
],
apis: [
{
type: 'category',
label: 'APIs',
items: [
{
type: "autogenerated",
dirName: "data/apis",
},
],
collapsible: false,
},
],
indexers: [
{
type: 'category',
label: 'Indexers',
items: [
{
type: "autogenerated",
dirName: "data/indexers",
},
],
collapsible: false,
},
],
oracles: [
{
type: 'category',
label: 'Oracles',
items: [
{
type: "autogenerated",
dirName: "data/oracles",
},
],
collapsible: false,
},
],
anchor_platform: ap_sidebars.anchor_platform,
stellar_disbursement_platform: sdp_sidebars.stellar_disbursement_platform,
};
Expand Down
5 changes: 0 additions & 5 deletions config/theme/navbar.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -359,11 +359,6 @@ const data: NavbarItem = {
label: 'Build Your Own',
activeBasePath: 'docs/data/indexers/build-your-own'
},
{
to: '/docs/data/indexers/indexer-providers',
label: 'Providers',
activeBasePath: 'docs/data/indexers/indexer-providers'
},
{
type: 'html',
value: '<hr><a href="/docs/data/oracles" class="subtitle"><small>Oracles</small>',
Expand Down
2 changes: 1 addition & 1 deletion docs/build/guides/basics/classic-transition.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ You can set up an RPC environment by hosting your own node or using a third-part

### Ingesting Smart Contract Events

Horizon offers an effects endpoint that describes state changes executed by classic operations. Similarly, contracts emit [events](../../../learn/fundamentals/stellar-data-structures/events.mdx) that describe changes to their state, which can be fetched via RPC's API. Off-chain solutions can [monitor and ingest these events](../../../build/guides/events/ingest.mdx) (for token transfers or protocol updates) and remain in sync with on-chain data. Each event is defined by the contract and is subject to standards applied to the implementation. Depending on the requirements for retention, a solution might have to handle ingestion directly or use a [third-party service](../../../data/indexers/indexer-providers/indexer-providers.mdx) for a longer-term history.
Horizon offers an effects endpoint that describes state changes executed by classic operations. Similarly, contracts emit [events](../../../learn/fundamentals/stellar-data-structures/events.mdx) that describe changes to their state, which can be fetched via RPC's API. Off-chain solutions can [monitor and ingest these events](../../../build/guides/events/ingest.mdx) (for token transfers or protocol updates) and remain in sync with on-chain data. Each event is defined by the contract and is subject to standards applied to the implementation. Depending on the requirements for retention, a solution might have to handle ingestion directly or use a [third-party service](../../../data/indexers/README.mdx) for a longer-term history.

### Simulating Transaction

Expand Down
2 changes: 1 addition & 1 deletion docs/build/guides/dapps/frontend-guide.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -744,7 +744,7 @@ This is made possible by using the `server.getEvents` method which allows you to

We will be editing the `CounterPage` component to read events from the counter smart contract immediately the page loads to get the initial counter value and update instead of using "Unknown". Before we continue, please take a look at the [contract code](https://github.com/stellar/soroban-examples/blob/main/events/src/lib.rs). In the contract code, an event named `increment` is emitted whenever the `increment` function is called. It is published over 2 topics, `increment` and `COUNTER` and we need to listen to these topics to get the events.

The topics are stored in a data type called `symbol` and we will need to convert both `increment` and `COUNTER` to `symbol` before we can use them in the [`server.getEvents`](../../../data/apis/rpc/api-reference/methods/getEvents) method. At maximum, stellar RPCs keep track of events for 7 days and you can query events that happened within the last 7 days, so if you need to store events for longer, you may need to make use of an [indexer](../../../data/indexers/indexer-providers).
The topics are stored in a data type called `symbol` and we will need to convert both `increment` and `COUNTER` to `symbol` before we can use them in the [`server.getEvents`](../../../data/apis/rpc/api-reference/methods/getEvents.mdx) method. At maximum, stellar RPCs keep track of events for 7 days and you can query events that happened within the last 7 days, so if you need to store events for longer, you may need to make use of an [indexer](../../../data/indexers/README.mdx).

To use events,we edit our counter page and add the following code:

Expand Down
1 change: 1 addition & 0 deletions docs/data/README.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
title: Data Overview
sidebar_position: 0
---

There are several products to choose from when interacting with the Stellar Network, and each one is optimized for different access patterns, so it's important to choose the right tool based on your specific use-case. These tools allow users to query network data, submit transactions, and interact with smart contracts without needing to understand the low-level details of Stellar Core.
Expand Down
1 change: 1 addition & 0 deletions docs/data/analytics/hubble/developer-guide/README.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
title: Developer Guide
sidebar_position: 15
sidebar_key: hubble-developer-guide
---

import DocCardList from "@theme/DocCardList";
Expand Down
111 changes: 105 additions & 6 deletions docs/data/indexers/README.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,113 @@
---
title: Indexer Overview
title: Indexers Overview
sidebar_label: Indexers
sidebar_position: 0
---

Learn more about indexing and indexers for Stellar network data.
## What is an indexer, and why would you need one?

## [Build Your Own Indexer](./build-your-own/README.mdx)
When you first start building on a smart contract platform like Stellar, at first you can get everything you need from RPC calls. You write a contract, or launch a Stellar Classic Asset and [wrap it in a smart contract](../../tokens/stellar-asset-contract.mdx), and you make a bundle of RPC calls from your frontend app directly to your smart contracts.

Building your own indexer with tools like Galexie allows you to export Stellar ledger metadata to external storage for custom data processing and analysis. This enables greater flexibility in structuring and querying blockchain data based on specific use cases.
At some point, though, you are likely to run into limitations with this approach.

## [Data Indexer Providers](./indexer-providers/indexer-providers.mdx)
1. You might need to make _too many_ RPC calls. Imagine: your app lists 20 of a user's NFTs per page. First RPC call: get a list of NFT IDs for this user. Then: 20 simultaneous RPC calls to get data for each NFT. And possibly: for each of those, make additional HTTP requests to S3 buckets or IPFS nodes to fetch image data, etc. This can make your app slow, or even get you rate-limited by an RPC provider.
2. You might want to show _historic_ data, like every time an NFT changed ownership. This sort of data lives _in the history of the blockchain_, but is not accessible via RPC calls to the most up-to-date version of a specific contract. Those would only return the current owner.

Data indexers process and structure blockchain data, making it more accessible and queryable for users with advanced analytics and visualization capabilities. They offer features like transaction flow analysis and DeFi metric tracking, providing a user-friendly and cost-effective alternative for exploring blockchain activity. Along with analytics, applications can use data indexers to track value movement of assets, track assets within an account, trace suspicious operations, and more.
Additionally, even some information that is currently available from [Horizon] will also require indexing services in the future, as Horizon will be deprecated.

## What exactly is indexing?

The word "indexing" has come to encompass a large array of use-cases. What all of them have in common is that they process and structure blockchain data. Let's think of it in the order of what you, an app-builder or small team, will need:

### 1. Off-the-shelf APIs for common data {#portfolio-apis}

This is the kind of stuff you might get from Horizon today, but can include more than that. For example, many indexing services in this category also provide the "every time an NFT changed ownership" kind of data mentioned above.

Lots of apps need this kind of data, and it tends to have a standard shape. This makes it profitable for companies to build SaaS products, offering API access for a price (often with a free tier, appropriate for your early-stage startup or hackathon project).

In the biz, these are called _Portfolio APIs._ Well-known companies offering them:

- [Alchemy]: most popular choice on Ethereum; now in talks to expand their service to Stellar, with a targeted launch in the first half of 2026.

- [Allium]: currently building Stellar support, launching Q1 2026

- [OBSRVR]: a home-grown Stellar-native offering, providing both RPC services and [Obsrvr Gateway](https://www.withobsrvr.com/products/gateway), promising "powerful APIs for real-time data fetching, transaction processing, and easy integration into existing systems."

- [Horizon]: as mentioned above, Horizon will soon be deprecated, but for now it is the only way to get some of this data.

### 2. Streaming & transforming data to a custom app database {#custom-transformations}

Not all apps have data that fits a standard shape. Most, probably, need custom data transformation.

Consider again the NFT example. Wouldn't it be great to return all needed NFT data in _one_ request, rather than N+1? That's what this style of indexing provides. And even better: you can (often) use these indexing solutions to aggregate data from multiple sources. If your NFT app stores additional information on IPFS, you could consolidate all of that off-chain data to your database as well. This makes your app much faster, saves you network & RPC requests, and creates new architecture possibilities.

Well-known options:

- [The Graph]: one of the earliest and most popular-at-the-time options on Ethereum. They now offer three main products:
1. [Subgraphs](https://thegraph.com/docs/en/subgraphs/developing/subgraphs/): their main & most famous offering, providing a [decentralized approach](https://thegraph.com/docs/en/subgraphs/developing/publishing/publishing-a-subgraph/) to custom data transformation & hosting. The Graph's use of GraphQL APIs made GraphQL a popular choice for this entire category of indexing.
2. **Token API**, for Indexing [Use Case #1](#portfolio-apis) described above
3. **Substreams**, for [Use Case #3](#analytics) described below

The Graph offers Stellar support for Substreams, with no current plans to expand Subgraph or Token API support to Stellar.

- [Goldsky]: one of the currently-most-loved options on Ethereum for this use-case. Goldsky provides [two main products](https://docs.goldsky.com/subgraph-vs-mirror):
1. **Subgraphs**: similar to those offered by The Graph, but data lives on Goldsky's own infrastructure rather than a decentralized network. Goldsky only offers this for EVM-based chains, with no plans to offer Subgraph support for Stellar.
2. **Mirror**, also called **Pipelines**: a highly efficient tool to Extract, Transform, & Load (ETL) data into your own database. Goldsky Mirrors already support Stellar; see [their documentation](https://docs.goldsky.com/chains/stellar).

- [Mercury]: a home-grown Stellar-native team providing streamlined Soroban (Smart Contract) support via their [Retroshades](https://docs.mercurydata.app/retroshades/introduction-to-retroshades) product. Note that this streamlined Soroban support comes at the cost of _only_ supporting Soroban! Mercury also provide [Mercury "Classic"](https://docs.mercurydata.app/mercury-classic/introduction), giving access to contract events & Stellar transactions via a GraphQL interface, which might fit Indexer Use Case #1 more.

- [SubQuery]: Decentralized Indexer SDK, Decentralized RPCs, & AI Apps. Supports 300+ chains. Like The Graph, uses a decentralized model.

- [OnFinality]: a big player in the Polkadot ecosystem, now expanding to other blockchains. OnFinality provides data _hosting_ services for your SubQuery logic. SubQuery is the _software_, OnFinality is the _infra_, hosting 1. the pre-transformed raw Stellar data, 2. your SubGraph SDK-authored Extract, Transform, and Load (ETL) processor and 3. your final transformed data.

- [Allium]: in addition to their Portfolio APIs offering, Allium is also under contract with SDF to build out tools for both Indexing Use Case #2 (this one) and Indexing Use Case #3 (see below), with target launch date of Q1 2026.

- [Space and Time]: one challenge with most indexing approaches is the reintroduction of trusted 3rd parties into what is otherwise a verifiable, trustless software stack. Space and Time aims to fix this with "Proof of Indexing" and "Proof of SQL", using Zero-Knowledge proofs to offer tamper-proof computation for enterprises and dapps. Space and Time's Stellar support [launched](https://www.spaceandtime.io/blog/space-and-time-enables-new-sophisticated-financial-apps-for-the-stellar-ecosystem) in Q4 2025.

- [OBSRVR Flow]: "Structured ledger data and contract events straight to your app or warehouse—no ETL needed." Currently in private beta.

### 3. Blockchain-Flavored Big-Data Analytics {#analytics}

For business intelligence, compliance, tracing suspicious operations, DeFi metric tracking, transaction flow analysis, etc. When your enterprise reaches a certain scale, it's worth paying Data Engineers to set up custom ETL pipelines and manage databases/datalakes, and then paying Data Analysts to answer questions about how people are interacting with your systems.

The companies building tools for Indexer Use Case #2 above (The Graph, Goldsky, Allium, etc) tend to also have tools for Use Case #3.

While often referred to as _indexing_, you can also think of this category as _analytics_. See the [analytics documentation](../analytics/README.mdx) for solutions custom-tailored to this use-case.

## Build Your Own

If none of the indexing providers mentioned above currently meet your needs, you can also build your own. Start with this [tutorial on how to build your own custom network ingestion pipeline](../../build/apps/ingest-sdk/overview.mdx). Along the way, you'll use the following tools & services:

### [Galexie](./build-your-own/galexie/README.mdx)

Galexie is a tool for acquiring Stellar ledger metadata from the network and exporting to external storage, a data lake. Galexie is the foundation of the Composable Data Pipeline (CDP) and serves as the first step in extracting raw Stellar ledger metadata and making it accessible. Learn more about CDP’s benefits and applications in [this blog post](https://stellar.org/blog/developers/composable-data-platform).

**Why Use It:**

- You want to maintain a data lake of pre-computed ledger metadata for historical and currently closed network ledgers.

### [Ingest SDK](./build-your-own/ingest-sdk/README.mdx)

A set of Golang packages which can be used within an application as a programmatic domain model to interact with Stellar network.

**Why Use It:**

- You want rapid development of applications in Golang which can acquire and parse ledger meta data and ledger entries from Stellar network.
- You want an intuitive, compile-time, type-safe application developer experience.
- You want to programmatically access History Archives to retrieve ledger entries.

### [Processors](./build-your-own/processors/README.mdx)

A suite of Go packages that help you parse Stellar blockchain data.

[Horizon]: ../apis/horizon/README.mdx
[Allium]: https://www.allium.so/
[Alchemy]: https://www.alchemy.com/
[Goldsky]: https://goldsky.com/
[OBSRVR]: https://www.withobsrvr.com/
[The Graph]: https://thegraph.com/docs/en/about/
[Mercury]: https://docs.mercurydata.app/
[SubQuery]: https://subquery.network/
[OnFinality]: https://onfinality.io/
[Space and Time]: https://www.spaceandtime.io/
[OBSRVR Flow]: https://www.withobsrvr.com/products/flow
4 changes: 2 additions & 2 deletions docs/data/indexers/build-your-own/README.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
title: Build Your Own Indexer
title: Build Your Own
sidebar_position: 0
---

Learn about the services and tools that you can build custom indexing with.
If none of the indexing providers mentioned [in the overview](../README.mdx) meet your needs, you can also build your own. Start with this [tutorial on how to build your own custom network ingestion pipeline](../../../build/apps/ingest-sdk/overview.mdx). Along the way, you'll use the following tools & services:

## [Galexie](./galexie/README.mdx)

Expand Down
1 change: 1 addition & 0 deletions docs/data/indexers/build-your-own/ingest-sdk/developer_guide/README.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
---
title: Developer Guide
sidebar_position: 0
sidebar_key: ingest-sdk-developer-guide
---

The Stellar [ingest SDK](https://github.com/stellar/go-stellar-sdk/tree/main/ingest) is a set of packages for retrieving and processing ledger metadata from the Stellar network. Developers can use this SDK to build custom ingestion engines.
Expand Down
8 changes: 0 additions & 8 deletions docs/data/indexers/build-your-own/tutorials/README.mdx
View file
Open in desktop

This file was deleted.

Loading
Loading