Suggestion: replace most calls to getProgramAccounts with lighter getTokenLargestAccounts

[oboxodo]

I have a new recommendation for a possible improvement in metaboss. Been wanting to actually make a PR for it for a couple weeks already but I'm facing reality and I don't have the Rust experience needed to do it in < 1 hour (yet) nor the time to spend more than that 😅. So I'll throw a suggestion in here for someone else to try it.

Reading a bunch of chats on different Discord servers (but mostly #rpc-support on GenesysGo's Discord) consensus seems to be that getProgramAccounts calls are very expensive.

I have my own TypeScript-based script to fetch a list of holders because I needed to do more than what metaboss does but I did my initial code looking at metaboss', so it follows a similar logic.

When getting a holders snapshot, the current code performs a getProgramAccounts call to fetch the list of all NFTs from a CM (or Authority). That's as best as it can get AFAIK. But then, for each NFT, it makes a new getProgramAccounts call to fetch the associated token account and grab the owner address from there.

I figured I could use getTokenLargestAccounts and grab the first element from the list instead of calling gPA. The downside is that this function only gets me the address of the associated token account. But I collect all the ATAs for each NFT, then split them in chunks of 50 each and make a single getMultipleAccounts call to get the rest of the data and then I can grab the owner from there.

I made those changes to my TypeScript metaboss-like code and it worked beautifully last week. I'm told by GenesysGo's support team that those calls are way lighter than gPA so I thought I'd suggest this as an alternative implementation. In total, I'm making a just few more RPC calls, but hopefully lighter ones.

Thoughts?

[alvinsga]

Have you managed to execute getProgramAccounts on the mainnet? I'm trying to get a snapshot of mints but it seems like the getProgramAccounts api has been disabled on the mainnet and even when using a custom RPC I get 504 Gateway Timeout. Works perfectly fine on the devnet though.

[oboxodo]

@alvinsga I understand some RPCs are currently timing out on gPA calls and the official api.mainnet-beta.solana.com is responding with "410 Gone" which seems to indicate they removed support for it.

Chatting on GenesysGo's Discord it looks like the timeouts could be a bug on Solana's latest 1.9.9 version. But I don't know anything official.

[samuelvanderwaal]

There's definitely issues with gPA calls on v1.9.9 for programs with large amounts of accounts (e.g. token-metadata). Not sure when it's going to get fixed but I'm going to look at using transaction crawling to get around this as well as implement @oboxodo's suggestion for snapshot hodlers performance improvements.

[oboxodo]

I understand this is related to the gPA problems: https://twitter.com/SolanaStatus/status/1501078159269244934

@samuelvanderwaal mind explaining just a bit of what you mean with "Transaction Crawling"? You got me curious.

[samuelvanderwaal]

I understand this is related to the gPA problems: https://twitter.com/SolanaStatus/status/1501078159269244934

@samuelvanderwaal mind explaining just a bit of what you mean with "Transaction Crawling"? You got me curious.

Transaction crawling just means getting all transactions associated with a specific account and then parsing them to find specific instructions and from there finding specific accounts in those instructions.

So in theory, I think you could find all transactions associated with a specific verified creator address, and then parse for the token-metadata program and create_metadata instructions and find all the created mint accounts from there. I haven't actually sat down and thought through all the design for this yet so might be missing some complications.

Here's a write up on how to do this specifically to find collection items: https://www.notion.so/metaplex/Get-Collection-Methods-1ff0b118e4ce4605971df60e753a8559

[oboxodo]

Ah, yeah. Understood. Such a PITA... :P

[samuelvanderwaal]

Ah, yeah. Understood. Such a PITA... :P

Indeed.

[samuelvanderwaal]

I have a new recommendation for a possible improvement in metaboss. Been wanting to actually make a PR for it for a couple weeks already but I'm facing reality and I don't have the Rust experience needed to do it in < 1 hour (yet) nor the time to spend more than that 😅. So I'll throw a suggestion in here for someone else to try it.

Reading a bunch of chats on different Discord servers (but mostly #rpc-support on GenesysGo's Discord) consensus seems to be that getProgramAccounts calls are very expensive.

I have my own TypeScript-based script to fetch a list of holders because I needed to do more than what metaboss does but I did my initial code looking at metaboss', so it follows a similar logic.

When getting a holders snapshot, the current code performs a getProgramAccounts call to fetch the list of all NFTs from a CM (or Authority). That's as best as it can get AFAIK. But then, for each NFT, it makes a new getProgramAccounts call to fetch the associated token account and grab the owner address from there.

I figured I could use getTokenLargestAccounts and grab the first element from the list instead of calling gPA. The downside is that this function only gets me the address of the associated token account. But I collect all the ATAs for each NFT, then split them in chunks of 50 each and make a single getMultipleAccounts call to get the rest of the data and then I can grab the owner from there.

I made those changes to my TypeScript metaboss-like code and it worked beautifully last week. I'm told by GenesysGo's support team that those calls are way lighter than gPA so I thought I'd suggest this as an alternative implementation. In total, I'm making a just few more RPC calls, but hopefully lighter ones.

Thoughts?

Did you do a direct comparison between this method and gPA? Mainnet and devnet, or just mainnet?

[oboxodo]

Did you do a direct comparison between this method and gPA? Mainnet and devnet, or just mainnet?

Went directly with mainnet and worked beautifully.

[samuelvanderwaal]

FYI, I did try this refactor and it actually slowed down the command because of the lack of an async Solana client. With 1.10 there's now a nonblocking client so I may give this another try at some point.