Provide a control for subgraph pruning in the subgraph manifest

[lutter]

With #3666, operators can control pruning of individual subgraphs, but it requires user action for each subgraph that should be pruned.

There should be a control that automatically prunes subgraphs without the need to explicitly run graphman prune. There are a few options for this:

• Make pruning part of the subgraph manifest so that subgraph authors can indicate how much history should be retained
• Allow operators to set how much history to retain through configuration, e.g., for each chain in graph-node.toml, or through additional rules in graph-node.toml based on subgraph names

Making pruning part of the manifest has the downside that subgraphs that only differ in how much history they retain will have different IPFS hashes, even though they are really the same subgraph, and can all be used to serve the same queries, assuming that the query is for a block height within the subgraph's available history. As an example, a subgraph that retains all history and a subgraph that only retains the last 1000 blocks of history can both be used to respond to a query at a block that is less than 1000 blocks behind the subgraph head.

[azf20]

That is indeed a drawback of the Manifest Hash as unique identifier. Presumably to change that pattern would be a very significant refactor? We have previously discussed identifying discrete and re-usable indexed regions of the manifest - and perhaps decoupling the indexing definition (datasources + startblocks + mappings + entities) from the subgraph definition would be a step in that direction, but that looks like a rabbithole.
It is relevant to other query only settings, which would also include changes to the Schema which don't impact indexing, for example the addition of derived fields.

In terms of indexer controls, I agree that wholesale and subgraph specific settings would probably be useful here.

[github-actions]

Looks like this issue has been open for 6 months with no activity. Is it still relevant? If not, please remember to close it.

[azf20]

Having discussed this "in the round", proposing that we allow for control by both subgraph developers and operators.

1. Add "history" to the subgraph manifest:

history: 10000

This is the clearest way for subgraph developers to radiate intent, but it creates limited flexibility and no scope for re-use.

Considerations:

• Should we make this configurable by network? i.e. history: - ethereum: 10000 (This goes against the fact that in practice each subgraph deployment must use the same network across data sources)
• What happens if an indexer wants to override the history for a given deployment? -> indexer configuration should take precedence in this case (and won't impact POI)

2. Let operators set default history per network in graph-node.toml

[chains.mainnet]
history = 10000

They can set individual history by deployment using graphman prune

Proposing the following order of preference when pruning an individual deployment:

1. Deployment configuration
2. Manifest
3. Graph node default

So if the indexer has applied graphman prune with a given block range, that should be applied. If no deployment configuration has been applied, and there is a manifest configuration, that should be applied. Otherwise the default should be applied, if present.

Pruning is a "one way door", and as such appropriate warnings should be provided when altering or overriding a pruning setting, and graceful explanation should be provided when increasing the history_blocks, to indicate that this will not work for existing block ranges, but future indexing will respect the new setting.

[lutter]

2. Let operators set default history per network in `graph-node.toml`

[chains.mainnet]
history = 10000

I am generally ok with what's proposed here, but a per-network setting is a very big hammer, and will probably not be all that suitable in most cases. I feel that a mechanism like the one in #4325 would be better. We could also make pruning something that you can set through [[deployment.rule]] though I feel that is overloading that construct and mixing two different purposes.

[azf20]

Understood - in that case proposing that we focus the scope of this issue to the manifest definition, agree that deployment.rule and chain-level config is maybe not the right place for this.

[incrypto32]

@leoyvens @azf20 Following up on this discussion here

There have been users who wants control on the pruning height (maybe to keep the ability to graft from x blocks behind the chainHead). Right now we do this manually for them when they ping us and ask to prune X blocks.

Why dont we have an optional history on prune and user can choose to provide it?

indexerHints:
 - prune:
     history: 10000

[leoyvens]

I get curious on what benefit this control gives them. How do they know they will need to graft 10k blocks behind the chain head, but not 11k blocks behind chain head?

But I do not oppose to having the history parameter, as long as it's optional, as you mention.

[leoyvens]

There is an optimization we should exploit from the knowledge that a subgraph will be pruned from deployment time. Right now, we create GIST indexes on reference columns (foreign keys):

graph-node/store/postgres/src/relational/ddl.rs

Lines 216 to 222 in 25c720a

	if immutable {
	let index_expr = format!("{}, {}", column.name.quoted(), BLOCK_COLUMN);
	("btree".to_string(), index_expr)
	} else {
	let index_expr = format!("{}, {}", column.name.quoted(), BLOCK_RANGE_COLUMN);
	("gist".to_string(), index_expr)
	}

I propose that pruned subgraphs always use a btree index, by updating the condition to be if immutable || pruned. For a couple of reasons:

1. We know that the subgraph will be pruned, therefore the block_range will not be very selective, so we can assume that a regular btree index will result in at least equivalent performance.
2. We've seen GIST be much larger than their BTree equivalent, so using btree here will be a significant factor in DB space savings.