-
Notifications
You must be signed in to change notification settings - Fork 864
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor(experimental): add accounts package #1855
Conversation
Overall this API looks well-organized from the documentation, but I didn't review the code yet. I was hoping to discuss the intention(s) a bit more first before reviewing. First and foremost, my knee-jerk reaction here is that this seems like a lot of boilerplate for what I can pretty easily do with
This is exactly the same as this: type MyAccountData = { name: string; age: number };
const myDecoder: Decoder<MyAccountData> = getStructDecoder([
["name", getStringDecoder({ size: getU32Decoder() })],
["age", getU32Decoder()],
]);
const myDecodedAccount = myDecoder.decode(myAccount.data); Similarly, Speaking of which, the whole concept of Sorry to be so negative here, but I'm struggling to see exactly why we need all of this. On the positive side, I dig
Although I have to say I think including the address in the type signature might be overkill as well. ie: let myEncodedAccount: Account<Uint8Array, "1234..5678">;
let myEncodedAccount: Account<Uint8Array>; It's worth mentioning that another positive here is the legacy In summary I think we can still add some nice dressing to the library as it relates to working with accounts with about half of this stuff or less. |
Thanks for the detailed review @buffalojoec! ❤️ Overall, I agree that this library is a light abstraction layer over the current solution — i.e. It seems we at least both agree on the following premise: There needs to be an account definition that's distinct from the one tight to the JSON RPC spec. Why? Because:
Now, let's look into the other parts of the API. I'll give you more details as to why I've added them and if you still think they are overkill, I'm more than happy to trim it down. I just want you to get the full picture before making that decision. Maybe AccountsMy reason for introducing that "Schrödinger's account" concept is purely because, when building dApp, it's a pain to not know the address of the missing accounts. You're receiving an array of type const myAccountsOrNull = await rpc.getMultipleAccounts(myAddresses);
const addressesOfMissingAccounts = myAddresses.filter((address, index) => myAccountsOrNull[index] === null); Whereas I think it's more intuitive to do: const myMaybeAccounts = await fetchEncodedAccounts(rpc, myAddresses);
const addressesOfMissingAccounts = myMaybeAccounts
.filter(account => !account.exists)
.map(account => account.address) That being said, I can understand that, the cost of adding and supporting this helper type is maybe not worth the benefit it provides. Decoder helperHere as well, I agree that the decoder helper is maybe a bit overkill. Although if we go for the API change proposed in #1865, it would fit nicely with the Also, your before/after comparison is not completely correct: // With the decode helper.
const myDecodedAccount = decodeAccount(myAccount, myDecoder);
// Without the decode helper.
const myDecodedAccount = {
...myAccount,
data: myDecoder.decode(myAccount.data)[0]
}; Fetch helpersHere, I think these helpers are important. Even if we don't greatly change the structure of the Even if we discard the fetch helpers and export the parse helpers instead, we end up exposing two account definition to the end-user which is not ideal: // With parsing helpers only.
const myRpcAccount = await rpc.getAccountInfo(myAddress);
const myAccount = parseEncodedAccount(myAddress, myRpcAccount);
// With fetching helpers that hide the parsing from the end-user.
const myAccount = await fetchEncodedAccount(rpc, myAddress); And it's even worse with In short, these helpers are the abstraction layer between the raw RPC account info and the one we want to maintain in this library. Address type paramThis library consistently offer a |
Well that's exactly the context I needed. As expected, well thought-out approaches and arguments. Some of these use cases on the dApp side weren't apparent to me. I think we're now mostly in agreement, then. Using the
As a standalone call it doesn't really make a lot of sense, but considered across I'm sure you put plenty of thought into that, though, so whatever you think is best on this part I'm good with. Side note: what's the TypeScript annoyingness-rating of something like this: // MaybeEncodedAccount[]
maybeEncodedAccounts.map(
if (account.exists) {
<ExistingAccountComponent lamports={lamports} data={data} />
} else {
<NullAccountComponent address={address} />
}
)
I went with |
Nice, I'm glad this helped add some context to the PR! 🙌 If I understand correctly we're left with two subjects to make a decision on: Maybe accounts and the replacement name of Maybe accountsI'm not too bothered about that one to be honest. I can see The pros are:
The cons are:
The equivalent of not using // (Account | null)[] created from an array of `addresses`.
accountsOrNull.map((account, index) => {
if (account) {
<ExistingAccountComponent lamports={account.lamports} data={account.data} />
} else {
<NullAccountComponent address={addresses[index]} />
}
}) Naming account.owner
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shit. I had a bunch of comments last week and forgot to commit them.
https://discord.com/channels/428295358100013066/1072928238076182589/1176212465441312830 |
I think my concern with this API would be the lack of support for If I want to fetch say a token account, my best bet would be to use Whereas if I want to fetch some other program account I'd want to use the I think it's potentially a bit confusing to have the mix of lower level |
I'm just here to cast a soft-vote in the direction of ‘every abstraction has a maintenance cost but some deliver more value than others.’ |
@mcintyre94 That's a great point that I didn't get to touch on in the description of this PR so thanks for bringing that up. Where serialisation should live on Solana is a big question. One that we still don't have a clear answer for. The contenders for this question are:
Right now, RPCs offer a little bit of both where most of the time you'd get encoded account data but every now and then you'd be able to get the decoded data depending if that specific program is supported by your specific RPC. This inconsistency is not something I'd like to bring over to this API. Let's fully embrace the first contender, and provide a consistent API, regardless of the program you're requesting and the RPC you're using. What does this mean concretely for accounts that can be parsed by RPCs like token accounts? Well, it means we'll provide fetching and decoding helpers just like we would for any program. const myTokenAccount: Token = await fetchToken(rpc, myTokenAddress);
const myMintAccount: Mint = await fetchMint(rpc, myMintAddress);
const myCustomDummyAccount: CustomDummy = await fetchCustomDummy(rpc, myCustomDummyAddress); |
@lorisleiva I think that's actually helped me understand how this library fits in a bit more. Is the idea here that you want to generate code like |
@mcintyre94 Yes, that's right! For each account, I generate helpers like |
@lorisleiva @mcintyre94 If we wanted to support it, but maybe not encourage it, we could roll a Regardless, this would allow us to use With this approach, you get all of the niceties of Loris' API without doing all the decoding on your app. Maybe you're doing some large operation where it makes sense to put the parsing load (especially for tokens under SPL) on the server, rather than your client. In this scenario, Note |
@buffalojoec The issue is, even for the small subset of programs it supports, the So we'll end up having to not only write parser code for each of these programs but also make sure we keep updating that list of parser when more programs are added to that list. IMO this Here again it's a case of, would supporting |
@buffalojoec @mcintyre94 I just had a nice chat with @joncinque who gave me permission to quote him on that. Jon said the Regarding performances, it seems RPCs don't even store the decoded data and only the token owner is indexed (because it was needed for the Solana Explorer at the time). Therefore, by using I'd very much like to cast my vote for "let's deprecate this thing". 😄 |
Except by pushing the parsing logic to the RPC (or whatever server layer) you don't have to ship every single parser, as JavaScript, to the client, which shrinks bundle sizes and makes websites load faster. |
@steveluscher Until we have a proper solution for RPC-side serialisation, we're gonna have to ship these decoders for 99% of the programs anyway. Let's make a consistent Client-side serialisation API instead please. 🙏 See my comment here. |
3874e38
to
13d22dd
Compare
We've had lots of nice discussions here but I'm a bit lost now when it comes to figuring out the next steps for this to be merged. I believe this is a good API that's going to elevate code gen. One that's close to how Umi handles accounts which has had lots of positive feedback. Is there anything anyone is completely against in this PR? |
To be completely honest I don't think I have a good feeling for what we're trying to do with the higher-level API so it's a bit tricky to give useful feedback. I think this is a good API to expose to developers, though I don't think Overall I'm in favour 👍 |
Thanks Callum, to give you more context this library will help me generate the following types and functions for a given program account: // Taking the Metadata account from Metaplex as an example.
type Metadata<TAddress> = Account<MetadataAccountData, TAddress>;
type MetadataAccountData = { ... };
type MetadataAccountDataArgs = { ... };
function getMetadataAccountDataEncoder(): Encoder<MetadataAccountDataArgs> { ... }
function getMetadataAccountDataDecoder(): Decoder<MetadataAccountData> { ... }
function getMetadataAccountDataCodec(): Codec<MetadataAccountDataArgs, MetadataAccountData> { ... }
function decodeMetadata<TAddress>(encodedAccount: EncodedAccount<TAddress>): Metadata<TAddress> { ... }
async function fetchMetadata<TAddress>(context, address: Base58EncodedAddress<TAddress>, options): Promise<Metadata<TAddress>> { ... }
async function safeFetchMetadata<TAddress>(context, address: Base58EncodedAddress<TAddress>, options): Promise<Metadata<TAddress> | null> { ... }
async function fetchAllMetadata(context, addresses: Base58EncodedAddress[], options): Promise<Metadata[]> { ... }
async function safeFetchAllMetadata(context, addresses: Base58EncodedAddress[], options): Promise<Metadata[]> { ... }
// For fixed-size accounts only:
function getMetadataSize(): number { ... }
// For PDAs only:
async function findMetadataPda(context, seeds): Promise<ProgramDerivedAddress> { ... }
async function fetchMetadataFromSeeds(context, seeds, options): Promise<Metadata> { ... }
async function safeFetchMetadataFromSeeds(context, seeds, options): Promise<Metadata | null> { ... } The thing I can live without for this code gen though is the |
I guess I'm on the fence for I understand adding conversions for Anyone who wants to use this accounts API and has god knows how large of an application built on
I won't argue to what specific magnitude moving all decoding to the client will affect bundle size and app performance, but there's plenty of people in the ecosystem who definitely will. I'm also starting to think people might get confused by the wording of Sorry to continue to be vague, but I don't have a specific solution in mind yet. I'm stewing on: A: Manually roll converters from B is dangerous for RPC v1, but it can definitely work in v2. |
Thanks Joe, I'm also on the fence with MaybeAccounts and I might just remove them at this point. Regarding First of all, the helper method is called Now, to tackle your proposed solutions. A) requires us to provide custom parsing logic for each supported program which will end up making your bundle size even bigger! Decoding on the new web3.js always rely on the same set of codecs being imported (struct, address, u32, etc.) so the likelihood of you already having these codecs on your app is very high. Not to mention you'll need them anyway to encode and decode your transactions. Not only that, but it will couple the new web3.js with program logic which would violate one of our core principles: "Program wrappers belong to their own packages". B) is essentially implementing RPC-side serialisation which is far from being ready. |
What about creating one single Because I'm envisioning something like this for flow: type MyAccountData = { name: string; age: number };
const myAccount: JsonParsedAccount<"1234..5678">;
// { parsed: { info: { name: 'joe'; age: 4 }, type: 'foo' }, ... }
const myDecoder: Decoder<MyAccountData> = getJsonParsedDecoder([
["name", getStringDecoder({ size: getU32Decoder() })],
["age", getU32Decoder()],
]);
const myDecodedAccount = decodeAccount(myAccount, myDecoder);
myDecodedAccount satisfies Account<MyAccountData, "1234..5678">; You could also follow similar patterns for One drawback that jumps out at me is there's no way to guarantee the type response from the RPC as |
No top level dependency changes detected. Learn more about Socket for GitHub ↗︎ |
e1e5b32
to
f46a9d7
Compare
bc80020
to
b76d13f
Compare
Alright, this is ready for another review. Significant changes are:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if (typeof rpcAccount.data === 'string') return getBase58Encoder().encode(rpcAccount.data); | ||
if (typeof rpcAccount.data === 'object' && 'parsed' in rpcAccount.data) | ||
return rpcAccount.data.parsed.info as TData; | ||
if (rpcAccount.data[1] === 'base58') return getBase58Encoder().encode(rpcAccount.data[0]); | ||
if (rpcAccount.data[1] === 'base64') return getBase64Encoder().encode(rpcAccount.data[0]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So a few things.
- There's nothing really ‘rpc-y’ about account data. The only RPC-ness here is the shape of the account (ie. the fact that the data is at
account.data[1]
). It would make this thing more general purpose if the ‘parseRpcAccount’ andparseAccountData
methods were separate, and the former made use of the latter. Another option is to eliminateparseRpcAccount
altogether and make people index into their own data. Another option is to createparseBase64RpcAccount
andparseBase58RpcAccount
et cetera. - The way that this function is written hard links both the base58 and base64 encoder implementations. This penalizes the person who only ever ‘decodes’
jsonParsed
accounts. If you end up doing step 1 (or eliminatingparseRpcAccount
altogether) then you can makeparseBase58AccountData
,parseBase64AccountData
, et cetera methods so the unused ones can get DCE'd.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting points raised. Commenting here to follow discussion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great points.
- The RPC naming was simply to signal we are going from an account definition provided by the JSON RPC spec to an account definition provided by the library.
- You're 100% right, I didn't think about the treeshake optimisation we could do here.
As a result I did the following:
- Split
parseRpcAccount
into 3 distinct functions:parseBase64RpcAccount
,parseBase58RpcAccount
andparseJsonRpcAccount
. - Created a private
parseBaseAccount
function that each of these use under the hood. - Split
fetchAccount
intofetchEncodedAccount
(which usesbase64
encoding) andfetchJsonParsedAccount
(which usesjsonParsed
encoding). - Similarly, split
fetchAccounts
intofetchEncodedAccounts
andfetchJsonParsedAccounts
. - Updated the tests and README accordingly.
Let me know what you think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since you're making the distinction from encoded
, should we also have a MaybeJsonParsedAccount
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
JSON parsed accounts can be returned as MaybeAccount<MyData>
directly. Do you mean that new type would be the union of this and MaybeEncodedAccount
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ohhh my b, I think I was confusing it with decode
. Lgtm.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lgtm @lorisleiva!!
I left two pointers to ongoing discussions I think should land before this goes in.
if (typeof rpcAccount.data === 'string') return getBase58Encoder().encode(rpcAccount.data); | ||
if (typeof rpcAccount.data === 'object' && 'parsed' in rpcAccount.data) | ||
return rpcAccount.data.parsed.info as TData; | ||
if (rpcAccount.data[1] === 'base58') return getBase58Encoder().encode(rpcAccount.data[0]); | ||
if (rpcAccount.data[1] === 'base64') return getBase64Encoder().encode(rpcAccount.data[0]); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting points raised. Commenting here to follow discussion.
9482bee
to
9ae24d5
Compare
9ae24d5
to
1034d79
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for all the committed revisions!
Because there has been no activity on this PR for 14 days since it was merged, it has been automatically locked. Please open a new issue if it requires a follow up. |
This PR adds a new
@solana/accounts
package that allows the new web3.js library to have its own definition of a Solana account as opposed to relying on the one provided by the RPC client that is limited to encoded accounts.See README below for more details.
This package contains types and helper methods for representing, fetching and decoding Solana accounts. It can be used standalone, but it is also exported as part of the Solana JavaScript SDK
@solana/web3.js@experimental
.It provides a unified definition of a Solana account regardless of how it was retrieved and can represent both encoded and decoded accounts. It also introduces the concept of a
MaybeAccount
which represents a fetched account that may or may not exist on-chain whilst keeping track of its address in both cases.Helper functions are provided for fetching, parsing and decoding accounts as well as asserting that an account exists.
Types
BaseAccount
The
BaseAccount
type defines the attributes common to all Solana accounts. Namely, it contains everything stored on-chain except the account data itself.This package also exports a
BASE_ACCOUNT_SIZE
constant representing the size of theBaseAccount
attributes in bytes.Account
andEncodedAccount
The
Account
type contains all the information relevant to a Solana account. It contains theBaseAccount
described above as well as the account data and the address of the account.The account data can be represented as either a
Uint8Array
— meaning the account is encoded — or a custom data type — meaning the account is decoded.The
EncodedAccount
type can also be used to represent an encoded account and is equivalent to anAccount
with aUint8Array
account data.MaybeAccount
andMaybeEncodedAccount
The
MaybeAccount
type is a union type representing an account that may or may not exist on-chain. When the account exists, it is represented as anAccount
type with an additionalexists
attribute set totrue
. When it does not exist, it is represented by an object containing only the address of the account and anexists
attribute set tofalse
.Similarly to the
Account
type, theMaybeAccount
type can be used to represent an encoded account by using theUint8Array
data type or by using theMaybeEncodedAccount
helper type.Functions
assertAccountExists()
Given a
MaybeAccount
, this function asserts that the account exists and allows it to be used as anAccount
type going forward.parseBase64RpcAccount()
This function parses a base64-encoded account provided by the RPC client into an
EncodedAccount
type or aMaybeEncodedAccount
type if the raw data can be set tonull
.parseBase58RpcAccount()
This function parses a base58-encoded account provided by the RPC client into an
EncodedAccount
type or aMaybeEncodedAccount
type if the raw data can be set tonull
.parseJsonRpcAccount()
This function parses an arbitrary
jsonParsed
account provided by the RPC client into anAccount
type or aMaybeAccount
type if the raw data can be set tonull
. The expected data type should be explicitly provided as the first type parameter.fetchEncodedAccount()
This function fetches a
MaybeEncodedAccount
from the provided RPC client and address. It uses thegetAccountInfo
RPC method under the hood with base64 encoding and an additional configuration object can be provided to customize the behavior of the RPC call.fetchEncodedAccounts()
This function fetches an array of
MaybeEncodedAccount
from the provided RPC client and an array of addresses. It uses thegetMultipleAccounts
RPC method under the hood with base64 encodings and an additional configuration object can be provided to customize the behavior of the RPC call.fetchJsonParsedAccount()
This function fetches a
MaybeAccount
from the provided RPC client and address by usinggetAccountInfo
under the hood with thejsonParsed
encoding. It may also return aMaybeEncodedAccount
if the RPC client does not know how to parse the account at the requested address. In any case, the expected data type should be explicitly provided as the first type parameter.fetchJsonParsedAccounts()
Similarly to the
fetchJsonParsedAccount
method, this method fetches an array ofMaybeAccount
from a provided RPC client and an array of addresses. It uses thegetMultipleAccounts
RPC method under the hood with thejsonParsed
encoding. It may also return aMaybeEncodedAccount
instead of the expectedMaybeAccount
if the RPC client does not know how to parse some of the requested accounts. In any case, the array of expected data types should be explicitly provided as the first type parameter.decodeAccount()
This function transforms an
EncodedAccount
into anAccount
(or aMaybeEncodedAccount
into aMaybeAccount
) by decoding the account data using the providedDecoder
instance.