NIP-90: Data Vending Machines

[pablof7z]

This PR introduces the concept of Data Vending Machines.

DVMs are a way of using nostr as a marketplace for compute.

The type of compute offered can be very broad, transcriptions, image generation, or nostr algorithmic feeds.

For example, for algorithmic feeds, this method allows the creation of thousands of specialized Service Providers that compute feeds in all sorts of weird and interesting ways, instead of relying on a single endpoint a client integrates with to generate a feed, this allows all clients to request algorithmic feeds from an unlimited number of providers.

The flow described in this NIP is extremely flexible to allow for data vending machines to offer their services in any way they choose (e.g. pay before processing, pay before getting results, pay after getting results, have a "balance" and get job results, or any model they choose)

Implementations

Client: Highlighter [https://dev.highlighter.com]

Fedi has built Replit extensions that use DVMs for code error-correction.

Service Providers: Nostr Data Vending Machine

There are two more Service Providers implementations I'm aware of that are not open source yet.

Also: bounties have been opened for data vending machines for non-nostr developers: https://replit.com/bounties/@Fedi/ai4all-build-a-nostr-1

Not addressed in this NIP

Encrypted job requests

Not to be included in the first draft of this NIP, but encrypted job requests should be added. For example:

• publish job requests with some useful metadata of the job (e.g., length of audio to be transcribed), service providers offer to do the job, the customer replies with a NIP-04-like encrypted job requested encrypted with the service provider's pubkey.

Viewable: https://github.com/nostr-protocol/nips/blob/vending-machine/90.md

[staab]

Depending on relay implementation, this might never get received, and the service provider might only have a missing event to go off of. Maybe #669 would help.

[cj-chua]

Some feedback from implementing this NIP in PhotoBolt

One of the brilliant parts of this NIP is the Job Chaining feature. To enable a UX where the client could just set a budget, publish a "Job Chain" and simply wait for service providers to complete the jobs, the client needs to know the total cost upfront. Proposing an ask field on kind 65000 job feedback event, where service provider would use this field to indicate the TOTAL cost for completing the job.

What is the mimetype for a URL? For example, a client might want the result of a generated image in the form of an url. I can't seem to find a mimetype for URLs

TLDR:

1. ask tag in kind 65000 that indicates total cost for the job
2. What should the mimetype for URL be?

[pablof7z]

Some feedback from implementing this NIP in PhotoBolt

One of the brilliant parts of this NIP is the Job Chaining feature. To enable a UX where the client could just set a budget, publish a "Job Chain" and simply wait for service providers to complete the jobs, the client needs to know the total cost upfront. Proposing an ask field on kind 65000 job feedback event, where service provider would use this field to indicate the TOTAL cost for completing the job.

The issue with that is that each step (each job) of a job chain can be served by different service providers, so each job needs to be priced individually. I think this should be addressed at the UX level of the client, where the client can ask for a total "allowance" and then have the intelligence to decide what the right split is (e.g. by looking at what other jobs have posted, or hardcoding, depending on the client)

[pablof7z]

What should the mimetype for URL be?

The idea of the mime-type in the output tag is the format the client expects, so if the service provider returns a URL where the output is accessible, then that file should be generated in the specified format.

I.e. an image generation with [ "output", "image/png" ] would be served by making a PNG file available in some URL the service provider generates.

Makes sense?

[fiatjaf]

Why not use some normal or ephemeral kind ranges?

Maybe ephemeral for the requests, normal for the responses since they are meant to be reused.

[fiatjaf]

Why is there a range of kinds for job requests, but not a range of kinds for job results? I think the opposite makes more sense.

Job results could be a range between 4001 and 4999, each representing a specific kind of job. These can referenced later if people want. If 4001 is a transcription someone can browse all transcriptions later or request relays for transcription events, for example. The kind difference is important because they may be used by all sorts of Nostr clients that are not using DVMs directly.

Job requests can have a single kind, since they're only going to be used by DVM-specific clients. They can be of kind 4000 and have a tag referencing their specific standardized type. For example, this would be requesting a transcription:

{
  kind: 4000,
  tags: [
    ["k", "4001"]
  ]
}

[cj-chua]

each job needs to be priced individually

Agreed that each jobs need to be priced individually.

However, for an individual job, the client can't infer the total cost the service provider is charging for the job, because amount tag in job feedback could be representing only the partial amount. Is this assertion correct? If so, the client couldn't provide a UX where the users get to see the total cost breakdown of a job / chain of jobs prior to execution, cause total cost for an individual job is not presented upfront by the service provider

.

an image generation with [ "output", "image/png" ] would be served by making a PNG file available in some URL the service provider generates

If a client wants the service provider to return an image in base64, because the base64 image will be used as an input to another job (only base64 is allowed as the input), the client can't accept job result from service providers that provide image results in the form of a url. How do we reconcile in this situation?

Seems to me that ["output", "image/png"] is not detailed enough? Client needs a way to specify if the result should be in the form of base64 or url etc

[cj-chua]

Maybe ephemeral for the requests

Doesn't ephemeral job request takes away the "marketplace nature" of this NIP? non-ephemeral events have the advantage where a job request residing on a relay may not have takers the first few days, but eventually picked up and processed after a relevant service provider comes online. Also, it takes away the opportunity for a client to show all outstanding job requests on the network in a specific period of time, for what it's worth

.

Why is there a range of kinds for job requests, but not a range of kinds for job results? I think the opposite makes more sense.

Job results could be a range between 4001 and 4999, each representing a specific kind of job. These can referenced later if people want. If 4001 is a transcription someone can browse all transcriptions later or request relays for transcription events, for example. The kind difference is important because they may be used by all sorts of Nostr clients that are not using DVMs directly.

Job requests can have a single kind, since they're only going to be used by DVM-specific clients. They can be of kind 4000 and have a tag referencing their specific standardized type. For example, this would be requesting a transcription:

{
  kind: 4000,
  tags: [
    ["k", "4001"]
  ]
}

This sounds interesting. The current NIP could allow easy-querying by generic clients by adding an additional job-request-kind in the tag for Job Feedback and Job Result, but at that point maybe it makes more sense to flip things around as proposed

I suppose Job Feedback stays the same under this proposal? Because if not there will be 2 kinds for each job type, one for Job Feedback and the other Job Result

[pablof7z]

Why is there a range of kinds for job requests, but not a range of kinds for job results? I think the opposite makes more sense.

Job results could be a range between 4001 and 4999, each representing a specific kind of job. These can referenced later if people want. If 4001 is a transcription someone can browse all transcriptions later or request relays for transcription events, for example. The kind difference is important because they may be used by all sorts of Nostr clients that are not using DVMs directly.

Job requests can have a single kind, since they're only going to be used by DVM-specific clients. They can be of kind 4000 and have a tag referencing their specific standardized type. For example, this would be requesting a transcription:

{
  kind: 4000,
  tags: [
    ["k", "4001"]
  ]
}

Yeah, ngl, while writing some of my clients I wished I had a way to query directly for the result like this kind-per-job-result allows. I ended up first querying for job-requests and then from there for job-results with the #e tags.

My main reason for having a kind per job requests was that it allows using NIP-89 so that users can signal "preferred bitcoin podcast transcription service", but if we add a k tag.

Lol, I actually went from a single kind + a j tag for the job type after @fiatjaf recommended going in this direction 😂

We could go with this modification of making a single kind + a k tag and still support NIP-89 by making a small modification to NIP-89 to accommodate for this.

[pablof7z]

Maybe ephemeral for the requests

Doesn't ephemeral job request takes away the "marketplace nature" of this NIP? non-ephemeral events have the advantage where a job request residing on a relay may not have takers the first few days, but eventually picked up and processed after a relevant service provider comes online. Also, it takes away the opportunity for a client to show all outstanding job requests on the network in a specific period of time, for what it's worth

Yeah, I considered ephemeral at first and discarded for this reason; it also grants an advantage to DVMs that have been running for longer and that can understand the market better because of this data, so it might be a bit centralizing.

.

Why is there a range of kinds for job requests, but not a range of kinds for job results? I think the opposite makes more sense.
Job results could be a range between 4001 and 4999, each representing a specific kind of job. These can referenced later if people want. If 4001 is a transcription someone can browse all transcriptions later or request relays for transcription events, for example. The kind difference is important because they may be used by all sorts of Nostr clients that are not using DVMs directly.
Job requests can have a single kind, since they're only going to be used by DVM-specific clients. They can be of kind 4000 and have a tag referencing their specific standardized type. For example, this would be requesting a transcription:

{
  kind: 4000,
  tags: [
    ["k", "4001"]
  ]
}

This sounds interesting. The current NIP could allow easy-querying by generic clients by adding an additional job-request-kind in the tag for Job Feedback and Job Result, but at that point maybe it makes more sense to flip things around as proposed

I suppose Job Feedback stays the same under this proposal? Because if not there will be 2 kinds for each job type, one for Job Feedback and the other Job Result

Yeah, job feedback stays the same; doesn't need extra kinds

[fiatjaf]

Can't we do a separate kind per job request + separate kind for job responses then?

20001 -> request translation
20002 -> translation response

20003 -> request transcription
20004 -> transcription response

...

[pablof7z]

My only hesitation would be using 2x the kind numbers, but other than that wouldn't have an objection with it.

[cj-chua]

@pablof7z Curious what you think about this? #682 (comment)

Were the questions clear?

[Semisol]

This NIP needs to be more clear.

[Semisol]

Which output?

[Semisol]

Can we define this more strictly?

[Semisol]

It would not make sense to force sats, especially when the NIP isn't tightly integrated with Bitcoin.

[Semisol]

The economic incentives of this NIP also make no sense.

[fiatjaf]

My only hesitation would be using 2x the kind numbers, but other than that wouldn't have an objection with it.

I think we're fine using 2x the kind numbers.

[cj-chua]

Considering that multiple NIP90 clients have came online, we should probably start finalizing some of the outstanding stuff

There seems to be consensus on the job-result kind modification (give a unique job-result kind to each job request)

I have created a PR here with the appropriate modification. Please let me know what you think

[starbackr-dev]

I agree with the results for event kinds 66001-67000. However, we've encountered challenges in dynamically managing the list of 65xxx job requests. The NIP provides examples like 65002 for podcast transcription and 65005 for image generation. Yet, these examples and number allocations pertain to very specific application topics. Should we consider exploring public lists where each application can display their supported job request types?

[starbackr-dev]

Thanks for the big refactor.. May I ask why the change to kind numbers? Any specific reason or just to make it simple? One thing I always do is experiment with the client to see how users use it. We now have DVM agent (kind: 65000) running inside Current App. Also have PlebAI agents working with kind:1 and Kind:4 events. We recorded over 5k interactions so far and one thing stood out very clear. Kind:4 events was used way more than others for a reason. users want anonymity as well as safeguard input and output. If I am creating a logo for my new business, then I don't want anyone to see it until I finalize it and release my site. So I added optional encryption to input params and output. The structure is very simillar to NIP-51 lists where you encrypt the tags and store it in the content field.

Here's the pull request. Please review and let me know for comments.

#823

[fiatjaf]

The idea behind standardizing kind numbers is that the results of DVM jobs can be queried and reused by others than the job requester.

If you want an encrypted thing then maybe we should have a single kind for "encrypted response" using the NIP-44 gift-wrap model or something like that?

[starbackr-dev]

The idea behind standardizing kind numbers is that the results of DVM jobs can be queried and reused by others than the job requester.

Agree.

If you want an encrypted thing then maybe we should have a single kind for "encrypted response" using the NIP-44 gift-wrap model or something like that?

It is not just the response but also the input params (Prompts and other specs) that need to be encrypted as well. Since the p tag is available on both request and result, we can use the content field to store the encrypted event. This is similar to NIP-51 private lists.

Don't want to gift-wrap the whole event b'cos we still need others to see that there was a DVM request and response between user and DVM agent but don't need to know the actual content. This is good for building DVM agent reputation that they completed x no. of events.

This PR captures both input and output encryption. #823

[cj-chua]

Here is a PR to fix the README
#824

Left a few comments. Looks good overall

[pablof7z]

the idea is that any app can very quickly see if there is a job result for the same input it would have provided, this might seem trivial, but for things where speed matters for UX, saving a round trip is important.

The example I have in mind is translation, which is the case where I see this having the largest immediate impact; instead of pinging a translation server one time per user for the same note ID, clients could query for translations of the event ids they want to see a translation for in a single REQ.

REQ, .., { kinds: [<translation-result-kind>], "#i": [ "event_ids"... ] }

[pablof7z]

I think we can define this at the kind-level, some (most?) kinds have a single output type (e.g. nostr content discovery is always a list of tags)

[cj-chua]

There is a potential for confusing the relay as the marker or the marker as the relay, given that both are optional.

How about making it mandatory that

• <relay> MUST be present for event or job input-type,
• <relay> MUST be absent for all other types that don't require querying relays

[pablof7z]

this is already convention per other NIPs that do the same, when wanting to leave relay empty and put a marker the relay should be an empty string. I can add a note to make this explicit here.

[pablof7z]

May I ask why the change to kind numbers? Any specific reason or just to make it simple?

NIP-01 changed to limit the maximum kind number to 16 bits, so the range 65000/65999 was now invalid; since I had to change the range I went for a lower range that FJ suggested and was free.

[starbackr-dev]

I am excited that this got merged but we cannot implement the changes without encryption. My feedback on Encryption and changes I requested was not addressed.

As developers we need to hear and implement user feedback if this NIP is going to be successful. We have heard from multiple people that they need the user prompt and results to be private. Hence I proposed an encryption similar to NIP-51 using the 'p' tag that is already present and 'content' field which is empty. We can move to NIP-44 encryption once that is audited.

@pablof7z please review the below PR and let me know why this was left out?

It is not just the response but also the input params (Prompts and other specs) that need to be encrypted as well. Since the p tag is available on both request and result, we can use the content field to store the encrypted event. This is similar to NIP-51 private lists.

Don't want to gift-wrap the whole event b'cos we still need others to see that there was a DVM request and response between user and DVM agent but don't need to know the actual content. This is good for building DVM agent reputation that they completed x no. of events.

This PR captures both input and output encryption. #823