Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NEXT] Adapt to UCAN 0.9 #532

Merged
merged 56 commits into from Jul 20, 2023
Merged

[NEXT] Adapt to UCAN 0.9 #532

merged 56 commits into from Jul 20, 2023

Conversation

icidasset
Copy link
Contributor

@icidasset icidasset commented Jul 5, 2023

This PR aims to bring the UCAN 0.9 to the SDK. The prototype version of UCAN we currently have is mixed into a lot aspects of the SDK. Because of this I have disabled and removed a lot functionality, such as most importantly, device linking, but more on that later.

After merging this, the next branch will be able to make somewhat functioning apps again. You can't make a Fission account yet though. You can load the file system without having an account and play around with it, like so:

import * as odd from "@oddjs/odd"

const program = await odd.program({
  namespace: "odd-test"
})

const fs = await program.fileSystem.load()

Goals of this PR

UCAN

Ideally we'd upgrade directly to UCAN v0.10. But we can't do that just yet (waiting on libs to upgrade), so I opted for 0.9 for now. That said, it should be quite straightforward to upgrade to the next version.

File System

The next branch is already using rs-wnfs primarily, but you can't make an app with it yet. This PR builds further on that and connects various pieces. Most importantly here is the private node mounting system. Which is where you take a private ref to load a private node using rs-wnfs. These secrets need to be passed from device to device in order to gain access to various parts of the private file system.

Linking

The SDK also has this delegation system (auth lobby). We've gradually abstracted this lobby system more and more, and this PR takes another step further. Because device linking and app-to-app linking is so similar, we've decided to merge the APIs and the channel over which the UCANs and secrets are transferred.

I've completed the initial API surface which you can see in various places, but it's not finished just yet. The idea is to do this:

// authority
program.authority.provide()
program.authority.on("query", ...)

// delegate
program.authority.request({
  from: ...,
  query: [ odd.authority.fileSystem("read", path) ]
})
program.authority.on("challenge", ...)

This'll be implemented using two components: authority & channel
Other things to check out: src/authority/query.ts

Accounts & Identifiers

The account system and identity layer have also been abstracted. We currently have usernames mixed in everywhere, this PR gets rid of that. Also very important, @hugomrdias had the great idea to always delegate from the identifier to the agent, which this PR implements. I'm using the naming "identifier" here instead of "identity" because that's @bmann his preferred naming which I agree with. Humans have multiple identifiers, not identities.

This makes implementing various identifier systems much easier. There will always be a delegation from the identifier to agent, and the agent delegates to the account system. Account and identifier systems can be implemented using their relevant components (account & identifier). The first account system to be implemented will be the new Fission server.

Registering an account has changed a bit because of the abstraction:

// Instead of passing the username directly, this now depends on the component implementation.
// But it'll always be an object you pass in.
const formValues = { username: "example", email: "..." }
const { ok } = await program.account.canRegister(formValues)
const { ok } = await program.account.register(formValues)

If either ok is false, there will be reason attached to it so you know why it failed.

You can also add additional methods to the implementation that should be made available on the program.account object. This is called the annex. So for example, the fission account implementation will most likely have a method to request a code via email which is required for the registration.

Depot

I've added a flush method to the Depot component. This is part of separating the data from the account layer. This should make it possible to for example use Web3Storage as the file system block storage (or in addition to Fission's storage).

The way this works is, you have two components that are called when a mutation is made in the file system (after a short debounce). You have the account system which is responsible for the data root and the depot, which is responsible for the IPLD block storage. Whenever the data root is going to be updated, flush is called. So you could use this to boot up your IPFS node in the browser, setting up a bitswap connection. Or to gather the changed IPLD blocks, package them up in a CAR file and push them to Web3Storage.

Cabinet

I've renamed the UCAN repository to the "cabinet". That is where your "ucan tickets" and access keys (file system secrets) are stored. The idea here is that this would be the place where you go look to check if you have access to a particular resource, be it related to your account or your file system.

New flow

With all these changes, the flow to work with a program has changed to:

// Root device, where you register your account
const outcome = await program.authority.isGranted()

if (outcome.granted) {
  // Already registered
  const fs = await program.fileSystem.load()

} else {
  // No access, could log this to find out what is missing.
  console.log(outcome.reason)

  // But for now let's assume we haven't registered yet.
  await program.account.register()

}

From here you'd link a device or provide partial access to another entity (as mentioned above).

Smaller Changes

  • Replaces user session objects with program.access.isGranted()
  • Removed the DID code, uses @hugomrdias his libraries
  • Removed keystore-idb
  • Removed DIDs from the Program object, these don't really make sense anymore. Agent DID should never be used anymore by the developer, always use identifier DID. Sharing/exchange DID needs to be rethought, will be implemented later together with private-data sharing.
  • Removed the crypto component, doesn't make sense anymore to have this as a component. Every JS language supports the web crypto API now anyway. Less references to pass around. Agent keys can still be customised using the agent component.
  • Reference component has been removed, functionality has been moved to other components or into their own component.
  • Username availability check is now done through DNS to avoid fetch errors in browser.
  • Cleans up various unused common code.
  • Account UCAN is no longer stored separately
  • Storage locations have changed so it will not cause a conflict with older versions of the SDK
  • Simplified the debugging config, debug and debugging config params have been merged.
  • Backwards compatibility with older Webnative clients has been removed.
  • Added pre-commit hook that formats all the source files, markdown and json.
  • File system is not loaded automatically anymore.

To do

In this PR

  • Finish up login behaviour/implementation
  • Update comments in index
  • Fix tests
  • Fix recursive imports

In other PRs

Things that are missing, need to re-implemented or still need to be fixed:

  • Update rs-wnfs
  • Integrate rs-ucan
  • Use iso-signatures to sign data instead of using custom code
  • Reduce bundle size (see Discord discussion)
  • Linking (access component implementation)
  • Extensions
  • Private data sharing
  • File system versioning
  • File system recovery
  • Unix FS tree (pretty tree)

@bgins
Copy link
Member

bgins commented Jul 5, 2023

Looking the linking API:

// authority
const program = await odd.program(config, "authority")

program.access.provide()
program.access.on("query", ...)

// delegate
const program = await odd.program(config, "delegate", {
  request: [ odd.access.fileSystem("read", path) ]
})

program.access.request()
program.access.on("challenge", ...)

Love how this is coming along, "request... access" reads great to me. ❤️

One question here is whether or not this supports a delegate requesting access from another delegate. In the past, @expede has mentioned that we would like to support multiple levels of subdelegation.

Would that work with this API? Does putting the request into the program initialization make it difficult to declare who we are requesting access from?

@bgins
Copy link
Member

bgins commented Jul 5, 2023

This makes implementing various identifier systems much easier. There will always be a delegation from the identifier to agent, and the agent delegates to the account system. Account and identifier systems can be implemented using their relevant components (account & identifier). The first account system to be implemented will be the new Fission server.

This hierarchy sounds great for supporting interoperability. 💯

Replaces user session objects with program.isConnected()

Nice! Like the "connected" naming.

Username availability check is now done through DNS to avoid fetch errors in browser.

Is there a risk for race conditions here? Say a user registers a name, and a second user comes along and wants the same name. If we are waiting for DNS to propagate, could the username check fail? This might be alright, as I imagine registration itself would fail.

Backwards compatibility with older Webnative clients has been removed

Yes, bon voyage 👋 . When we have a rough timeline for next, we should announce this in Discord to give folks a heads up.

@icidasset
Copy link
Contributor Author

One question here is whether or not this supports a delegate requesting access from another delegate. In the past, @expede has mentioned that we would like to support multiple levels of subdelegation.

@bgins Yeah good point. I don't know if you saw the recent discussion about that on Discord. But I'd like some well-defined use cases for this, otherwise I don't know really how to shape the API. But maybe we opt to make it generic instead?

I'd like to avoid roping the initialisation into this, but not sure how. With these changes when you run your program in authority mode, you get a register and login function on your program. Because, based on the lobby model, you would only register an account or do device linking in the lobby (the authority), not the "delegate". Then there's also the question of how isConnected() should function, it'll need different functionality based on the mode it's operating in (authority or delegate). Right?

There's something here about specific use cases. We assume there's an "authority" which requires full read & write access to the file system and a "delegate" which could operate with a very small subset of that (eg. only read portion of file system). But is that always the case? 🤷‍♂️

Maybe we could do something like this instead?

const program = await odd.program(config)

// authority
await program.account.register()
await program.account.login()

await program.account.isConnected()
// NOTE: Could check if UCAN has been delegated with full write access and if we can update the data root.
//             Also should potentially check if the configured account system has all the capabilities it needs.

// delegate
await program.access.request({
  from: ...,
  query: [ ... ]
})

await program.access.isGranted(query)

Thoughts?

Is there a risk for race conditions here? Say a user registers a name, and a second user comes along and wants the same name. If we are waiting for DNS to propagate, could the username check fail? This might be alright, as I imagine registration itself would fail.

Blaine is setting up some DNS magic which will make it look up the values in the database directly, so propagation will be faster I assume. That said, I think it'll be fine too, the registration itself would indeed fail.

@bgins
Copy link
Member

bgins commented Jul 6, 2023

But I'd like some well-defined use cases for this, otherwise I don't know really how to shape the API. But maybe we opt to make it generic instead?

Yeah, it would be nice to have use cases. Not sure what they are, and I can see how this makes it more challenging to design an API.

I'd like to avoid roping the initialisation into this, but not sure how. With these changes when you run your program in authority mode, you get a register and login function on your program. Because, based on the lobby model, you would only register an account or do device linking in the lobby (the authority), not the "delegate".

I'm agreed that there is a fundamental difference between an authority and a delegate, and I think it's a good design to only expose what one or the other needs.

Then there's also the question of how isConnected() should function, it'll need different functionality based on the mode it's operating in (authority or delegate). Right?

Are there cases where a delegate could start with a local-only file system that later gets merged into a file system granted by an authority? If so, I think isConnected() works for both. I think this works for offline use cases, where a delegate app may want to store some data then later request access to persist and sync it.

The way these would function differently is the method of requesting authority. The authority gets it by registering an account, and the delegate gets it from the authority.


const program = await odd.program(config)

// authority
await program.account.register()
await program.account.login()

await program.account.isConnected()
// NOTE: Could check if UCAN has been delegated with full write access and if we can update the data root.
//             Also should potentially check if the configured account system has all the capabilities it needs.

// delegate
await program.access.request({
  from: ...,
  query: [ ... ]
})

await program.access.isGranted(query)

Yes, I think this has potential. It would make the request for access more flexible. It might be possible to retain the distinction between authority and delegate? An authority will never need to request access, and a delegate won't need to register or login.

@bgins
Copy link
Member

bgins commented Jul 6, 2023

I was discussing a use cases for multiple levels of subdelegation with @QuinnWilton today, and I think we have a good example.

Suppose we have the following apps:

  • An auth lobby (an authority)
  • Diffuse, a music player (a delegate)
  • Notify, another music player (a delegate)
  • MetaFixer, an app that applies metadata to music files (a delegate)

Diffuse and Notify request access from the auth lobby to access public/music and public/notify, respectively. These are the directories where they store music.

MetaFixer is a secondary app that is accessible from Diffuse and Notify. It doesn't do much on its own, but on requesting access from Diffuse or Notify, it will look up metadata and store it with your music.

Diffuse and Notify have a button that starts linking directly with MetaFixer, subdelegating access to the music directories they have access to.

Alternatively, the user could be redirected to the MetaFixer where they request permission from the auth lobby. The disadvantage to this approach is the user loses context when they request access from the auth lobby.

Also, MetaFixer could request access to directories that it doesn't need. The user may not be aware that MetaFixer is asking for more, whereas requesting from Diffuse or Notify guarantees MetaFixer can only request what it needs.

Note also that MetaFixer will get access on an as needed basis. A user can start by applying metadata to music in public/music for Diffuse without access to public/notify. Later, they can request access through Notify for public/notify. In each case, they retain context for the access they are granting.

@QuinnWilton
Copy link

👋 I just read through this thread too, and I have one more thought on a possible use-case for sub-delegation:

We assume there's an "authority" which requires full read & write access to the file system and a "delegate" which could operate with a very small subset of that (eg. only read portion of file system). But is that always the case? 🤷‍♂️

I actually don't think this is necessarily always the case, even if in the current world it is! Baked into this sentence is the assumption that WNFS is the only service for which an authority may delegate access to, but there's a possible future where apps built using OddSDK may integrate with other UCAN enabled services too, for things like message buses, email, or SMS.

If these services are third-party, then the Fission auth lobby isn't likely to serve as an authority for all of them, and so some applications may require delegations from multiple different authorities. This means that it actually becomes possible for individual applications to hold more capabilities than any individual authority, and so the ability to sub-delegate from that rights-amplified application reduces the friction involved in sharing those capabilities with other applications.

In practice, this might show up as something like a Heroku or AWS style dashboard that accumulates capabilities from disparate, possibly third-party services, and can then further attenuate those capabilities to other applications.

Anyway, this is just me spitballing about another instance where sub-delegation shows up, and I understand that this isn't how things work today, so this isn't me suggesting that it's a use-case to be targeted by you in this refactor.

@icidasset
Copy link
Contributor Author

I was discussing a use cases for multiple levels of subdelegation with @QuinnWilton today, and I think we have a good example.

Thanks you two! And for picking an example I can relate to ❤️ Sounds like a great use case.

This means that it actually becomes possible for individual applications to hold more capabilities than any individual authority, and so the ability to sub-delegate from that rights-amplified application reduces the friction involved in sharing those capabilities with other applications.

Makes sense yeah. I'm actually getting very close to this, so I'll start working in that direction for the API 👍

@icidasset icidasset marked this pull request as ready for review July 14, 2023 18:15
@bgins bgins self-requested a review July 14, 2023 18:45
src/index.ts Outdated Show resolved Hide resolved
@bgins
Copy link
Member

bgins commented Jul 18, 2023

Removed DIDs from the Program object, these don't really make sense anymore. Agent DID should never be used anymore by the developer, always use identifier DID.

I think this information could still be useful to developers for understanding authorization flows. They may want to log the agent DID to track which agent they are working in, and we might want to show this information in the browser extension. We will eventually extend the browser extension to display UCANs, so this information will be good context for that as well.

File system is not loaded automatically anymore.

Yep, agreed it will be better to require developers to handle this explicitly. 👌

src/index.ts Show resolved Hide resolved
Copy link
Member

@bgins bgins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have looked through all the components. A bit more to go, but getting there. 😃

src/components/account/implementation/local-only.ts Outdated Show resolved Hide resolved
src/ucan/lookup.ts Show resolved Hide resolved
src/components/account/implementation/fission-base.ts Outdated Show resolved Hide resolved
src/components/account/implementation/fission-base.ts Outdated Show resolved Hide resolved
src/components/account/implementation/fission-base.ts Outdated Show resolved Hide resolved
src/components/account/implementation/fission-base.ts Outdated Show resolved Hide resolved
src/components/account/implementation/fission-base.ts Outdated Show resolved Hide resolved
src/components/agent/implementation/web-crypto-api.ts Outdated Show resolved Hide resolved
src/index.ts Show resolved Hide resolved
src/components/manners/implementation.ts Outdated Show resolved Hide resolved
@icidasset
Copy link
Contributor Author

@bgins Added back the DID shorthand methods, except for the sharing one, because that still needs to be figured out.

Copy link
Member

@hugomrdias hugomrdias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM left a few comments but nothing blocking we should merge asap and work of small PRs

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should look into using https://github.com/sindresorhus/emittery for a more robust event emitter

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may not be worth adding a dependency if we don't need the additional functionality.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding some additional context. We considered using Emittery when we initially implemented the event emitter.

Our thought process was: This is simple enough. We can implement it ourselves and save on bundle size while reducing the risk of supply chain attacks by having one less dependency.

Our plan was to extend it as needed, but at a certain point it may not be worth our time and effort.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with that approach with a few exceptions one being event emitter because they are very very tricky and are prone to memory leaks and uncaught exceptions.
emittery defends against a bunch of weird stuff ppl do inside handlers and adds a couple of nice to haves, it net positive IMO.

https://pkg-size.dev/emittery its small and has 0 deps.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be down for this change as well.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, let's change it. One request, we should make sure the type information is not degraded making the switch.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should i add did dns resolver to iso-did for this ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Up to you. Would it allow you to customise how the DNS resolving is done?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i would do DoH by default with round robin over like 5 options and allow for optional custom resolver

Comment on lines +21 to +30
canRegister: (formValues: Record<string, string>) => Promise<
{ canRegister: true } | { canRegister: false; reason: string }
>

/**
* How to register an account with this account system.
*/
register: (formValues: Record<string, string>, identifierUcan: Ucan) => Promise<
{ registered: true; ucans: Ucan[] } | { registered: false; reason: string }
>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we normalize the return types to a generic Result type { result?: T, error: Error }

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hugomrdias a bit of discussion around this here: #532 (comment)

Comment on lines 148 to 178
const response = await fetch(Fission.apiUrl(endpoints, "/user"), {
method: "PUT",
headers: {
authorization: `Bearer ${token}`,
"content-type": "application/json",
},
body: JSON.stringify(formValues),
})

if (response.status < 300) {
return {
registered: true,
ucans: [
// TODO: This should be done by the server
await Ucan.build({
audience: identifierUcan.payload.iss,
issuer: await Ucan.keyPair(dependencies.agent),
proofs: [Ucan.encode(identifierUcan)],

facts: [{ username }],
}),
],
// TODO: We need some UCANs here. We should get capabilities from the Fission server.
}
}

return {
registered: false,
reason: `Server error: ${response.statusText}`,
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should extract code like this to a fission server client and also export from that odd sdk components

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea yeah. Not sure if we'd ever make use of that outside of the SDK though, since the CLI will be written in Rust.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it allows for easier and faster iteration on server logic without messing with the SDK and we extract Fission related components

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can already iterate on that without messing with the SDK though, thanks to component system:

odd.program({
  account: fissionAccountComponentWithAdjustments
})

Comment on lines -107 to +116
}).then((response: Response) => {
method: "PUT",
}).then((response: Response): { updated: true } | { updated: false; reason: string } => {
if (response.status < 300) dependencies.manners.log("🪴 DNSLink updated:", cid)
else dependencies.manners.log("🔥 Failed to update DNSLink for:", cid)
return { success: response.status < 300 }

return response.ok ? { updated: true } : { updated: false, reason: response.statusText }
}).catch(err => {
dependencies.manners.log("🔥 Failed to update DNSLink for:", cid)
console.error(err)
return { success: false }

return { updated: false, reason: err }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these return types should be normalised IMO and should return something other than a boolean, either cid or an Error. Especially reason should only be of one single type.

Comment on lines +35 to +44
/**
* This goes hand in hand with the DID `keyTypes` record from the crypto component.
*/
keyAlgorithm: () => Promise<KeyType>

/**
* The JWT algorithm string for agent UCANs.
*/
ucanAlgorithm: () => Promise<SignatureAlgorithm>
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are these async ? they should be pretty much constants

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just keeping all the options open, doesn't hurt since all the functions that depend on these are async anyway.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its creating micro tasks in a bunch of places to return a constant string

/**
* DID associated with this agent.
*/
did: () => Promise<string>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this async ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same answer as with the agent methods.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why didnt you use the RSA Signer from iso-signature here ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah, totally forgot about that. I wrote this before you released that, will change soon.

Comment on lines +23 to +27
return {
did: async () => DIDKey.fromPublicKey("RSA", exportedKey).toString(),
sign: async data => WebCryptoAPIAgent.sign(data, signingKey),
ucanAlgorithm: async () => "RS256",
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here why not just use Signer ?

Copy link
Member

@bgins bgins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic work on this! So much good stuff in here. ✨

I'm enjoying how the new auth and file system APIs are shaping up. It's looking great!

@icidasset icidasset merged commit ec327a8 into next Jul 20, 2023
@icidasset icidasset deleted the icidasset/ucans branch July 20, 2023 10:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants