Cannot add large schema to data model

[fhirfly]

Describe the bug
HTTP request to 'http://localhost:7007/api/v0/streams' failed with status 'Internal Server Error': {"error":"request entity too large"}

To Reproduce
Try to load a large schema. Mine is 4MB

Expected behavior
A clear and concise description of what you expected to happen.
The schema should load

Screenshots
node ceramic-model.mjs
file:///C:/Users/richb/Projects/glazin-fhir/node_modules/@ceramicnetwork/common/lib/utils/http-utils.js:21
throw new Error(HTTP request to '${url}' failed with status '${res.statusText}': ${text});
^

Error: HTTP request to 'http://localhost:7007/api/v0/streams' failed with status 'Internal Server Error': {"error":"request entity too large"}
at fetchJson (file:///C:/Users/fsdfsdf/Projects/glazin-fhir/node_modules/@ceramicnetwork/common/lib/utils/http-utils.js:21:15)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Function.createFromGenesis (file:///C:/Users/sdfsdf/Projects/glazin-fhir/node_modules/@ceramicnetwork/http-client/lib/document.js:42:27)
at async CeramicClient.createStreamFromGenesis (file:///C:/sdfsdf/sdfsdf/Projects/glazin-fhir/node_modules/@ceramicnetwork/http-client/lib/ceramic-http-client.js:46:24)
at async ModelManager._createDoc (file:///C:/Users/sdfsdf/Projects/glazin-fhir/node_modules/@glazed/devtools/dist/datamodel.js:148:16)
at async Promise.all (index 0)
at async ModelManager.createSchema (file:///C:/Users/sdfsdf/Projects/glazin-fhir/node_modules/@glazed/devtools/dist/datamodel.js:343:40)
at async file:///C:/Users/fsdfs/Projects/glazin-fhir/ceramic-model.mjs:26:22

Ceramic versions

Machine, OS, browser information (please complete the following information):

running daemon on linux
running client on windows 11, using the following depends:
{
"dependencies": {
"@ceramicnetwork/http-client": "^2.3.0",
"@glazed/devtools": "^0.2.0",
"dids": "^3.2.0",
"key-did-provider-ed25519": "^2.0.1",
"key-did-resolver": "^2.0.5",
"uint8arrays": "^3.0.0"
}
}

Additional context
Add any other context about the problem here.

[stbrody]

Yeah, ceramic commits are limited to 256kb. 4MB is very large for JSON text - why do you have such a large schema?

[fhirfly]

It’s not the commit that’s failing . It’s loading the schema. The data model is for FHIR, the international stanard for health data. It’s large because, well, health data is a big thing.

[stbrody]

I see, thanks for the information. I've filed a feature request to support larger documents in Ceramic, but there's a few different things that would have to change to make this work. The HTTP request size limit that you noted is one, but there is also the Ceramic commit size limit, as well as possibly the IPFS block size limit, that would all need to be dealt with. I'll leave this ticket open for now and we can use it to report any update towards increasing the Ceramic document size limits.

[fhirfly]

@stbrody What about the schema size limit on the server? This is an express setting, not something particular to ceramic. I tried setting it in the daemon config, but I wasn't exactly sure how to do it or if your Carmic's embedded express would lpick it up there. I would like to see us at least tackle a schema load. I would then worry about the size of commits. Data can be chunked, and the current stream might be just fine. Does the commit size constrain the size of the schema?

[stbrody]

Does the commit size constrain the size of the schema?

Yes, it does. Schemas are stored as documents in Ceramic same as any other data.

[fhirfly]

IPFS says the block size limit is 1MB, but obviously a file can be written to more than 1 block. Must Ceramic commits exists in a single block or can they be over more than 1 block?

[stbrody]

IPFS says the block size limit is 1MB, but obviously a file can be written to more than 1 block. Must Ceramic commits exists in a single block or can they be over more than 1 block?

I think that questions is exactly one of the things we would need to investigate as part of the work to enable larger documents on Ceramic

[stbrody]

@fhirfly, could you take your large FHIR schema and break it up into multiple smaller schemas for different parts of the overall data?

[fhirfly]

Unfortunately, the schemas are all related. Did you ever make the limitation go away?