RFC: ESIP-8 - Single Ethscription Creation through Blob Transactions

[tunnckoCore]

Let the hype begin.

I'll update it later today. We talked a lot today in #general, in #technical and in DM but.. yeah. Gotta get some sleep and formulate it.

Premise

Make it possible to create ethscriptions through Blob Transactions.

1. it should support raw files, data URIs, gzipped raw files, and gzipped data URIs? Or?
2. requires ESIP-6 , or something else in the calldata of the transaction - easier tracking across all blobs; no need to become even bigger surveillence system track every single blob too, on top of the every single transaction
3. requires eip4844 (type 3) transaction
4. support creation only for single ethscription (no batch)

If multiple blobs are found on a single ESIP-4844 transaction, they are combined to form a single ethscription. Current max, for some reason, is around 240-250kb (roughly 2 blobs). This might be because other are also filling the block.. which again, for some reason the max is 6 blobs per block, not 16 as supposed (probably temporary). Question is, how to know how many blobs there gonna be in a block before submitting the creation transaction, LOL.. so, anyway.

About the default uniqueness requirement

The blob hashes already support uniqueness check by default, eg. if the content is the same, the blob_versioned_hash is the same too. The calculation of this "blob hash", for short, is by hashing the kzg_commitment with SHA256, removing the first 2 characters and prefix with 0x01 - voila, you have the blob hash. Now, how to get commitments? Easy, can derive them from the data/content/hex "submitted" by users/transaction. Anyway, i'll post some more later.

moar on blobs

The splitting into blobs is done automatically. Eg. if the file is 200kb, it will make 2 blobs of 128kb, the last one will have empty bytes at the end and you gotta be careful with that and clean them to be able to have correct and proper content_uri. Waste few hours on that yesterday :D I'll share everything. Or it's somewhere in the discord.

more later

these are some "Blobbed Ethscriptions" i track while I was talking the whole day ;d

// Transactions with blobbed ethscriptions

//  0x53bb90b706f6fcbec44aade06ebc7205aeb7aec2a879d922b6875d4365968b9a - middlemarch first blob eths (shall be encipherd)
//   0x8b5953eb496dee0075a71155678d45385d21ff410683f889ab507282a8e2ebc5 - mine (top 5 coingecko, $73k btc)
//  0x228c0ecfe26a43e89411bafd04f10286cbf245108821e0eb597abefe8e3acb0b - mine (battle budiez #3) & ethscription (2 in 1)

//  0x012960b1466b47f154779c5cce3b17c819cd3e1ccbac56d7082576bc908c3e67 - blobbed $BLOB erc-20 (cryptpotr)
//  0x01490bf8de50ffb2b5817e591d6351b917fdc03304a15c0aefea5c069bd0ea13 - blobbed $mfpurrs erc-20 (cryptpotr)


// - Dancun Epoch 269568, Slot 8626176, Block 19426587


// BlockScout API

// - https://eth.blockscout.com/api/v2/main-page/indexing-status
// - https://eth.blockscout.com/api/v2/stats - prices, gas
// - https://eth.blockscout.com/api/v2/blobs/0x01490bf8de50ffb2b5817e591d6351b917fdc03304a15c0aefea5c069bd0ea13
// - https://eth.blockscout.com/api/v2/transactions/0x6a55ad2b21b4261dfb9312ba40df98041bf9d447efc4f55c57eb18868cdcbe3f

Also made a basic viewer https://blobs.wgw.lol

[RogerPodacter]

Nice!

I agree that we want a "normal" calldata ethscription that points to the blob. Such a "pointer" ethscription could even be reused to point to non-blob things! Here's my idea:

data:application/vnd.esc.content.metadata+json,{"location": "blobs://self", "type": "image/png", "length": 1234, "encoding": "...optional eg deflate, enables compression schemes that don't have magic bytes...", "sha256": "abcde", "extra-metadata": { "description": "optional metadata"}}

Or with the JSON splayed out:

{
    "location": "blobs://self",
    "type": "image/png",
    "length": 1234,
    "encoding": "...optional eg deflate...",
    "sha256": "abcde",
    "extra-metadata": {
        "description": "optional metadata"
    }
}

• This is a generic pointer that can point to "the concatenated blobs on this transaction" with the URI blobs://self.

• Because the blobs are raw bytes, we specify the content type.

• We also must specify the length because of the null byte padding at the end. There's no generic rule we can use to find the end of the content and the start of the padding, so we need the user to tell us.

• Encoding gives us the ability to support all kinds of compression, not just those with magic bytes like gzip.

• We need the content sha because the blob sha is the sha of the overall blob including padding. Having the actual content sha will enable future generations to verify the content even in the case the blob is lost. Indexers MUST verify this sha matches the actual content's sha at the time of indexing.

• Users can put whatever they want in extra metadata. They could even add a low-res preview of the blob-based content for more future proofing.

Indexers should have a separate field for the content the ethscription is pointing to and the data URI content of the ethscription itself. The rule that all ethscription content must be a data URI persists. Likewise uniqueness / ESIP-6 are applied to the data URI as normal. The presence of the content sha will ensure correct behavior.

Thoughts?

[RogerPodacter]

Questions:

How long must indexers store the blob data? 18 days?

Can you reference blobs in other transactions from your transaction? Not sure why you'd want to do that exactly; maybe to add commentary.

Should you be able to reference other ethscription content in your ethscription? Can you reference something that references something else? I'd prefer not to open a whole recursion can of worms as part of the blob change.

[tunnckoCore]

From the end to the top.

Does this user know the versioned hash at the time the create the tx? If so they could use that in the URI

Yes. I think so. In the JS side, we create the blob with toBlobs, which returns the hex of the data, padded; then we can use blobsToCommitments, which returns the kzg commitment and from there we can generate the blob hash; All that before sending the transaction, so yeah.

Maybe we call these "attachments" and ethscriptions can (eventually) have an array of them

I like metadata more.

How long must indexers store the blob data? 18 days?

Not sure. Thing is that there are 3 things - indexer, db, and API. In theory, if they are separate, Indexer don't need to know and store the actual data, it can ask the DB through the API. If they are not, yeah, the indexer will/should store it permanently.

Can you reference blobs in other transactions from your transaction?

Don't think so.

Should you be able to reference other ethscription content in your ethscription?

Where? From the metadata or from the blob's content? In anyway, i don't think we need any special/internal recursion thing. Recursion is easiest done with standardized api endpoints, and through html+js, so that you don't need anything special.

I agree that we want a "normal" calldata ethscription that points to the blob. Such a "pointer" ethscription could even be reused to point to non-blob things! Here's my idea:

I agree that we need a metadata in the calldata. BUT, don't agree with most of the points there. It's a waste of money and precious and more expensive calldata space, cuz most of the things can be inferred.

It should be as simple as possible, and light on indexer to choose to implement and support it.

This is a generic pointer that can point to "the concatenated blobs on this transaction" with the URI blobs://self.

What's the point of that? It will always be blobs://self, it's a waste on gas.

Because the blobs are raw bytes, we specify the content type.

We don't. Lets be strict and have the 90% of cases handled, the rest can be added through other ESIPs if such a need come at all (eg the docx example). It's also safer to have only a limited set of content types supported.

We also must specify the length because of the null byte padding at the end. There's no generic rule we can use to find the end of the content and the start of the padding, so we need the user to tell us.

We can do that too. It's not that hard. For most types you can see where the thing ends. I did it, it may not be guaranteed but it should work in most cases especially if we have only a limited content type support.

But that's 1 of the 2 things i can agree to include.

Encoding gives us the ability to support all kinds of compression, not just those with magic bytes like gzip.

Useless. Compression support for different things should be accepted from other ESIP anyway. If it's not a "required" field, okay. We can actually move it to the extra metadata field which should also be optional.

We need the content sha because the blob sha is the sha of the overall blob including padding. Having the actual content sha will enable future generations to verify the content even in the case the blob is lost.

Agree. It allows when people try to create ethscrition from content to check if this thing is "ever existed" if so, it won't be a valid creation, despite the fact it already don't exist.

Thing is, they will always exist. That's the point. So there's no such possibility, and may not even need this sha as requirement.

Indexers MUST verify this sha matches the actual content's sha at the time of indexing.

Why? You're decoding the whole thing into "proper content_uri" anyway, you'll have the actual content and make the sha and then check it against the DB if it exists or not and reject the creation if it already exist.

---

In general, turns out we don't need most of the stuff. I can agree for length, sha, and extra. Because the SHA will make the ethscription data pass the uniqueness check.

Likewise uniqueness / ESIP-6 are applied to the data URI as normal

It will be for the ethscription data, but you will need to check if the blob's content exists or not.

[tunnckoCore]

What about that:

The calldata won't be "ethscription", but 0x01<blob_hash>2c<length>, (2c = comma) or maybe not comma but something else (Glyph is using commas ;d).

0x01c2a37b23a653e7997744c0fa5d2e24d5b44808317ee0c0a7097ac08457f9e0 2c 2002

It's optimal and it's enough. Also.. killing the extra metadata here (maybe some other esip in future), is cool cuz

1. we won't pollute the ethscriptions with useless metadata stuff
2. less room for mistakes and shits; i'm fan of not allowing people to do a whole lot of stuff wrong.
3. Blob hashes always start with 0x01 and is easy to detect.
4. In this case we would have length required, to make calldata more unique and more easy to track; And because i want to make blob-only mini protocol, which transfer thing can work like sending transaction hashes in ethscriptions.

Glyph has similar format of calldata. This allows me to decrease the cost 2-10x of creation and transferring of tokens (cuz it's primarily calldata-based tokens), than ethscriptions and even facet.

Like 0x01<amount>, is "mint amount"; or 0x02<to_address>2c<amount> transfer to address; Seems like OP_CODES and ABI.

Calldata is so cool.. it's mind blowing. A lot of people don't realize how powerful it is. Cuz reality is the whole network is driven by this single thing LOL.

---

edit: in case of more than 1 blobs, it format will be mostly the same

0x01c2a37b23a653e7997744c0fa5d2e24d5b44808317ee0c0a7097ac08457f9e0 2c 2002 3b 01c2a37b23a653e7997744c0fa5d2e24d5b44808317ee0c0a7097ac08457f9e0 2c 3003

[RandolphJiffy]

Questions:

How long must indexers store the blob data? 18 days?

Can you reference blobs in other transactions from your transaction? Not sure why you'd want to do that exactly; maybe to add commentary.

Should you be able to reference other ethscription content in your ethscription? Can you reference something that references something else? I'd prefer not to open a whole recursion can of worms as part of the blob change.

We should permanently store blob data in the indexer, right? User data is precious.

[RogerPodacter]

We should permanently store blob data in the indexer, right? User data is precious.

Ethscriptions.com surely will, but it will be indexer by indexer. I think of it as "enhanced IPFS" in a way.