explorer: Use Unified Token-list API #201
Assignees
Comments
jacobcreech
changed the title
ngundotra
changed the title
|
@jacobcreech can you provide some examples where the current behavior is incorrect, or unacceptably slow? |
|
Explorer currently uses @solana/spl-token-registry which has a scary large bundlesize of 1.1MB gzipped, adding up to 23s on slower network speeds. Doing these legacy token-list lookups on a backend is much faster. Searchbar token logos seems to have since been removed when searching. Coingecko as a first choice when looking up metadata can be added to unified token-list as well, removing issues with certain protocols trying to update their now locked token metadata and pushing liability to coingecko. |
|
Will come back to this once we've migrated the website to Next JS |
mcintyre94
added a commit
that referenced
this issue
Jul 18, 2023
This PR refactors the search bar to remove the client-side legacy web3js dependency. This means that the core app layout, shared by every page, no longer pulls in legacy web3js - The client-side call to `getDomainInfo` is replaced with a GET request to a new API `/api/domain-info/[domain]` - This API just calls `getDomainInfo` and returns the response as JSON. This function and everything else in `utils/name-service` that can be run on the server are extracted to a new `utils/domain-info`. This is necessary because `utils/domain-info` has `use client` (needed for its client-side hooks) - The API has a 24h cache-response header. I haven't done any manual client-side caching, Next's fetch de-dupe should be sufficient. - For now I haven't changed how we get the domain info. Longer term I'd like to support multiple domains: #271 - I've extracted everything used by the search bar from `utils/tx` to a new `utils/programs`. I've removed the legacy web3js dependency, which was just used to pull in program public keys The result of this is that some pages load ~100kb less JS. I'm not exactly sure how some of these numbers work though, eg the homepage is already not including web3js despite including the search bar. <img width="1366" alt="Screenshot 2023-07-18 at 15 23 19" src="https://github.com/solana-labs/explorer/assets/1711350/a1cc43d8-1f76-44ca-a558-1f9f8a8d2f3d"> We can also inspect the bundle before and after this change, this is the app/layout endpoint. On master we have the legacy web3js <img width="2387" alt="Screenshot 2023-07-18 at 15 01 10" src="https://github.com/solana-labs/explorer/assets/1711350/fd0d4a6e-4e47-400a-b45f-fed2719415ef"> After we've stripped many of these dependencies out: <img width="2388" alt="Screenshot 2023-07-18 at 15 02 33" src="https://github.com/solana-labs/explorer/assets/1711350/04294a1c-03d7-4b67-8fcd-549cbc5a6215"> Mostly it shows that we should stop including that token list though... #201
mcintyre94
added a commit
that referenced
this issue
Jul 31, 2023
This PR removes and replaces the `@solana/spl-token-registry` dependency, which was by far the largest dependency and was pulling in an >3MB outdated JSON file on every page. It should improve performance across the board, especially on slower networks. Token data will also be much more up to date, and tokens created since we deprecated the token registry will be represented correctly. The main replacement for this dependency is [Solflare's Unified Token List (UTL) SDK](https://github.com/solflare-wallet/utl-sdk). This is a typescript SDK that provides information about a token. It's got great performance and covers most use cases we have really well. It has some tradeoffs: - It's quite a heavy dependency, mostly because of its `metaplex-foundation/js` dependency which allows it to fetch on-chain data for tokens that aren't in its API/CDN already. For this reason I've used the underlying API directly for the search bar, to avoid pulling the SDK into the root layout. This is actually the same behaviour as the SDK has for search anyway, since falling back to an on-chain query can't be done for search. - It doesn't have all the per-token data from the old token registry, specifically for extensions it only has the coingecko one, and is missing eg bridge contracts and websites. We only use these on the token address page. Longer term we might want to remove these since we don't have a way to get this data for new tokens. But for now I'm using a CDN version of our last token list to fetch the token info including these legacy fields. For tokens that aren't in that list I use the SDK as a fallback, and just return the data without those extensions. Other than these cases, we now use the SDK for all token info. Some specific implementation decisions: - The address/[address] page fetches token info once (using the legacy CDN, then API fallback), and uses prop drilling for child components. - `useAccountOwnedTokens` now returns tokens with the fetched name/logo/symbol for each, fetched from the UTL SDK. - The `<Address>` component used to search the token registry in all cases, like it does with programs etc. This is no longer the case since we no longer have the entire token list on every page. A new prop `tokenListInfo` can be used to pass known token info in. This is used for example when rendering the user's owned tokens. Alternatively `fetchTokenLabelInfo` can be used to asynchronously fetch the token for the address, and update the address to display it after fetching. This should be used for addresses that might be a token, for example transaction accounts. This is the equivalent of `useMetadata` which fetches NFT metadata. - I've used SWR in a few places where we fetch token info as an enhancement, usually just to display the symbol for a token as a suffix for a count, eg 5 USDC. In these cases I use the loading state of SWR to distinguish between data not fetched yet (we display no suffix) and no data available (we typically add the 'tokens' suffix) The result of all this is that the huge tokenlist file which dominated our bundle is gone: Master: <img width="2386" alt="master-all-min" src="https://github.com/solana-labs/explorer/assets/1711350/335135f5-7750-4962-8149-5fe542dd3cbd"> This branch: <img width="2387" alt="rtr-all-min" src="https://github.com/solana-labs/explorer/assets/1711350/da3b5c42-65fd-4121-8cf7-e2a8bc858047"> Looking at just the layout shared by all pages, the effect is even more drastic. Master: <img width="2388" alt="master-layout-min" src="https://github.com/solana-labs/explorer/assets/1711350/3e843dd0-7fc4-4c21-9a57-36b95bb393d8"> This branch: <img width="2388" alt="rtr-layout-min" src="https://github.com/solana-labs/explorer/assets/1711350/748f6fdc-d16f-4600-b840-6b72848e3ff6"> Closes #201 --------- Co-authored-by: steveluscher <me+github@steveluscher.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem
Currently the explorer queries the following to get token-metadata:
This has the following problems:
getProgramAccounts(nor recommended)Proposed Solution
Solflare created a Unified Token API that manages caching, searching, and verifying tokens via coingecko, legacy token-list, and Metaplex Fungible token metadata.
Proposal is to use it to create a faster experience for users on the explorer searching and viewing their tokens.
We have two options:
Proposing we use option 2 for speed to ship. Happy to work on it if consensus is met.
cc: @steveluscher @jstarry
The text was updated successfully, but these errors were encountered: