Setting Up Spellbook for the Next 1000 Spells 🚀🧙‍♂️

[couralex6]

TLDR: we are moving some token related models to a separate dbt project to increase the stability and scalability of Spellbook. Some spells will change location, but your workflow will mostly remain unchanged.

Hello Wizards! 🧙‍♂️

Thanks to the amazing work of our community, Spellbook will soon reach a size of 3000 spells! While we’re very excited about the growth of the project, we need to ensure that the project is properly set up to scale in the future. 🚀

As you may know, Spellbook is powered by DBT, which lets us manage the orchestration of those 3000 spells (DBT models). We love DBT and have no immediate plans to move away from it. However, the current single-project setup isn’t well suited for the size of Spellbook. As the project grew, we started noticing undesirable side effects like non-linear increase in compile times and pre-run introspection steps, slow CI runs, and very slow docs pages (to name a few).

To address this scale issue, we decided to pilot a multi-project layout that should facilitate the growth of Spellbook. We considered many approaches to split out the current DBT project and decided on the following:

• We will start by spinning out the upstream lineage of tokens_erc20 and tokens_native. These models are at the root of the largest sub-graph (sub-dag) in Spellbook. Removing them results in a series of smaller sub-graphs that are easier for us to manage.

• These models will live in a “sub-project” called tokens, which will live in a folder at the root of the Spellbook repo. All other models will remain in the same location for now. Removing those models from the main DBT project means that we won’t be able to read from them with the ref() function. Instead, we’ll have to use source() to read from them, and they’ll be treated as an external table. We already modified the children of the models removed to reflect those changes so you don’t need to.

• Since we are keeping the main sub-project at the root level, the development process will remain the same for the majority of the models. However, if you modify a model in the tokens project and wish to compile it locally, you’ll have to run dbt deps from the tokens/ directory before you run dbt compile. If you only rely on the CI to develop and test your models there are no changes to your development process!

In the future, we will move more models to their own sub-projects, but we first want to experiment with this directory layout and sort out potential issues before moving more models.

🚀🧙‍♂️

[MSilb7]

Nice! Just to confirm, the main functionality difference is that:

• Since we read these tables as "source," then the only spells that can write to tokens, must live in that tokens sub-project

  • i.e. if I have a spell that writes to tokens_erc20, then that has to live in the tokens sub-project (don't expect this for tokens, but maybe for other sectors).
  • But if we have a spell like "read from tokens, do some other stuff, then create a new table," that ~could live anywhere (i.e. from tokens, check against bridge contracts to identify if this token is L2-Native, or comes from the L1 bridge).

• Does this change also enable the "all tokens" table, so that symbols/decimals don't have to be manually curated anymore?

[couralex6]

Exactly, all spells writing to tokens will have to be in the tokens subproject. This is the exact list of spells being moved:

$ dbt -q ls -s +tokens_erc20 +tokens_native --resource-type model --output path
models/aave/aave_v3_tokens.sql
models/ovm/optimism/ovm_optimism_l2_token_factory.sql
models/the_granary/optimism/the_granary_optimism_tokens.sql
models/tokens/arbitrum/tokens_arbitrum_erc20.sql
models/tokens/avalanche_c/tokens_avalanche_c_erc20.sql
models/tokens/base/tokens_base_erc20.sql
models/tokens/bnb/tokens_bnb_bep20.sql
models/tokens/celo/tokens_celo_erc20.sql
models/tokens/tokens_erc20.sql
models/tokens/ethereum/tokens_ethereum_erc20.sql
models/tokens/fantom/tokens_fantom_erc20.sql
models/tokens/gnosis/tokens_gnosis_erc20.sql
models/tokens/tokens_native.sql
models/tokens/optimism/tokens_optimism_erc20.sql
models/tokens/optimism/tokens_optimism_erc20_bridged_mapping.sql
models/tokens/optimism/tokens_optimism_erc20_curated.sql
models/tokens/optimism/tokens_optimism_erc20_generated.sql
models/tokens/optimism/tokens_optimism_erc20_transfer_source.sql
models/tokens/polygon/tokens_polygon_erc20.sql
models/tokens/zksync/tokens_zksync_erc20.sql
models/tokens/zora/tokens_zora_erc20.sql
models/yearn/optimism/yearn_optimism_vaults.sql

The main thing to keep in mind when choosing where to place a spell is that we can't have any cyclical dependencies between sub-projects. Models in the Spellbook subproject can read from models in the Tokens subproject (as a source) because Tokens is upstream of Spellbook. On the other hand, models in Tokens should never read from models in Spellbook. With that said, we can move more models to Tokens in the future if we see any lineage issues.

[couralex6]

Does this change also enable the "all tokens" table, so that symbols/decimals don't have to be manually curated anymore?

This table will be released at a later time I believe. @jeff-dude

[jeff-dude]

• Since we read these tables as "source," then the only spells that can write to tokens, must live in that tokens sub-project

yes, think of each sub-project as the "final" spell of a lineage. tokens subproject = tokens.erc20 spell & everything upstream that builds it. once in a stable spot, we will look at having a dex and nft subproject, which will host all spells that build any spell in schemas dex or nft

• Does this change also enable the "all tokens" table, so that symbols/decimals don't have to be manually curated anymore?

the new automated tokens source will be a fast-follow after we launch tokens subproject, just to make sure we can get that out clean prior to introducing new variables. once automated source is live, most of existing lineage will be deprecated. you could query the raw token data now if you want, to test out data quality, prior to launching a spell on top:

    select
      i.blockchain,
      t.*
    from
      dune.definedfi.dataset_tokens t
      inner join evms.info i on t.networkid = i.chain_id

[couralex6]

PR #5260 was merged successfully! There are some open PRs that need conflicts resolved (I'll reach out on the affected PRs).

[tomfutago]

Hey guys, with this change - now ALL sources.yml are in the new ./sources directory, NOT just sources related to tokens, correct? So with any new spell I need to split sources.yml from all the other spell related files in 2 separate locations: ./sources & ./models? hmm, seems like an overkill..

[jeff-dude]

sources is one piece that i'm not fully confident on the new design. thinking out loud here, but maybe the best setup is:

• ./sources directory to host:
  • ./sources/_base_tables/ only raw base source (transactions, traces, etc. per chain), that are shared across subprojects
  • ./sources/_subprojects/ for spells built and intended to be used as sources in other subprojects, i.e. tokens.erc20 and all the upstream lineage to build it
• all other sources in their normal locations

i believe the one issue comes down to some sources needed in more than one subproject, especially as that grows into more. the downside is listing the source in multiple dbt projects. the upside is keeping all files in one subproject for a lineage (sources.yml, schema.yml, <model>.sql)

[tomfutago]

Yep, that'd be much cleaner - rather than having a mirror of full on directory structure in 2 different locations. not sure if that makes things easy to maintain and for contributors to find their way around it.

And don't get me wrong - i'm fully onboard with splitting off tokens to its own project to scale this beast!

[jeff-dude]

@couralex6 as the one who got their hands dirty on the sources stuff here, what do you think of my proposed setup above? am i missing something that makes the design break?

[couralex6]

@tomfutago Both dbt projects spellbook and tokens will read sources from the sources/ directory. No need to "split" sources across models/ and sources/.

If you place a source.yml file under models/, that source will only be accessible from within the spellbook project (which is totally fine).

The idea behind the sources folder was to have a central shared location for sources for all future subprojects, and that all sources would live there. With that said, I'm open to suggestions for more user friendly design!

With that said, I've been experimenting with a dbt package called dbt-codegen that can generate source yaml automatically with a connection to our metastore. Wizards wouldn't be able to run this locally, but i'm working on a way to run this from the CI and automatically generate source yml files. I'll try pushing a prototype by the end of the week!

[tomfutago]

Thanks @couralex6, that context around options and usage for sources makes much more sense to me now, I missed that info before somehow.

And looking forward what will come out of dbt-codegen testing - automating additions of sources.yml would be cool to have.

[jeff-dude]

for those following along, our next steps will be:

• finalize design after living in it and learning along the way (for example, sources convo above)
• replace tokens lineage of manual additions with new automated tokens source, likely complimented with some manual additions to account for any gaps in the automated source
• find next candidates for sub-projects (dex, nft, etc.)

we don't have exact timeframes on next steps here, as we will be putting a lot of focus on revamped labels the next few months.

[jeff-dude]

another proposal:
remove a layer of the directory path in new sub-projects.
before: tokens/models/tokens/arbitrum/tokens_arbitrum_erc20.sql
after: tokens/models/arbitrum/tokens_arbitrum_erc20.sql

remove extra token, as we know from root directory.

[couralex6]

Agreed, i'll do this tomorrow