How should RustGIS projects store their data? A proposal for the .rgis format

[Mavrick-F]

When you save a project in RustGIS, the .rgis file stores your maps, styles and layouts, and it also stores most of your layers' data. That keeps projects easy to send, but on real data the files get large, slow to open and memory-hungry. Some grow past the size RustGIS can open, so they can't be saved at all.

I'd like to fix this before 1.0. After 1.0, people will rightly expect saved projects to keep working, so the format should be settled first. This post sets out the problem with measurements, the choice it raises (should a project hold its data, or only point to it?), the design I'd prototype, and what else was considered.

The problem

A .rgis is one uncompressed JSON document. Every layer loaded whole is written out coordinate by coordinate. I measured realistic projects built from public data (TIGER/Line, Microsoft buildings, NHD, GNIS) with the v0.5.0 CLI:

Project	.rgis today	same JSON, zstd	Reopens?
Field survey (CSV, GPX, GeoJSON)	34 MiB	4.6 MiB	yes, 0.3 s
Watershed (NHD 0801)	114 MiB	32 MiB	yes
Shelby County (TIGER layers, buildings, 3 tool outputs)	341 MiB	95 MiB	yes, 4.6 s, 823 MiB RAM
Statewide (Tennessee roads and buildings, US counties, GNIS)	646 MiB	117 MiB	no: over the 512 MiB limit

A single step can cross the limit. A 30 m buffer of Shelby County's 76,666 TIGER edges made a 592 MiB project. Opening a project takes 2.4–3.9× its size in RAM, and autosave rewrites the whole file every five minutes. For scale, the Windows installer is 34 MiB.

Save used to write these files and report success, then fail to reopen them. #1364 makes Save refuse instead, which avoids a broken file but still leaves the work unsaved.

Fixes that don't need a decision

• fix(engine): reject projects too large to reopen before saving #1364, plus an error that names the size, the limit, and which layers to stream or export.
• Recovery copies written compressed. They're internal, so this raises no compatibility question.
• GeoParquet exports written with zstd and without the per-feature bounding-box columns. That makes them 2–4.5× smaller.
• The byte-based memory budget in Tracking: large data (layers over 2 million features, slow hillshade and GeoTIFF writing) #1111, and browser recovery, each on its own track.

The real question: what is a project?

In v0.1.0 a project was one file that held everything. A few hours later, #71 (for #66) streamed large GeoPackage, GeoParquet and FlatGeobuf files so they didn't have to fit in memory. Since then each format has added its own exception, so today whether a layer lives in the project depends on its format and size. Users can't predict it, and that's how the projects above outgrew the limit.

There are two coherent alternatives:

• The project points to data. This is the traditional model, the way QGIS and ArcGIS Pro work. The project holds maps, styles and layouts, and every layer refers to a file, database or service. Tool results go to real data files. You share by packaging the project with its data.
  • Upside: project files stay tiny, there's no size limit, and it's familiar.
  • Downside: a project is several files, links break, outputs pile up, and every handoff needs a package step.
  • Effect on the format: most of the storage problem goes away. Compressed JSON for the project plus GeoPackage for outputs would do.
• Self-contained by default. This is v0.1's idea, made to scale. Every layer is either in the project or linked, and you always know which. Add links, Import copies, tool results are in the project, and editing linked data is explicit (Edit Source, or make a project copy).
  • Upside: one file to work on and send.
  • Downside: the file has to store data efficiently and open without loading everything, and copies can drift from their sources.
  • Effect on the format: this is what the prototype below is for.

Either model changes what layer.addData does today, so the choice needs its own decision. Under either model, a team sharing data needs:

• named connections instead of paths, so a shared layer resolves on every machine, with connection definitions the team can share and credentials kept per person in the system keychain;
• copies that record their source, so RustGIS can say when a copy is out of date;
• notice when a linked source's schema changes, not just its rows;
• read-only shown up front, so Edit Source on a layer you can't write fails clearly instead of at save time.

If your team works with shared data, what else would it need?

The two models in more detail, and a team sharing data under each

The project points to data.

• For:
  • tiny project files and instant saves;
  • no size limit beyond the data's own;
  • shared data is the default;
  • data stays in standard formats;
  • the project is readable, diffable JSON;
  • most GIS users already know it.
• Against:
  • several files per project, so links break when only part of it is moved or emailed;
  • users must manage where outputs go;
  • every handoff needs a package;
  • unsaved scratch layers can be lost;
  • personal outputs pile up on shared drives;
  • the browser has no file paths, so its data would need private browser storage.

Self-contained by default.

• For:
  • one file to work on and send;
  • personal analysis stays tidy inside the project;
  • linked shared data still works: Add links it, and Edit Source writes back through the pending-edits model RustGIS already has for PostGIS and GeoPackage.
• Against:
  • the file must store data well and open lazily;
  • sharing a result is an explicit export;
  • imported or portable copies can drift from their sources.

A team sharing data. The canonical data belongs in a database: PostGIS, MySQL or SQL Server. RustGIS already edits PostGIS with pending edits, transactions and row-version checks, so two people can't silently overwrite each other. Read-only reference data can sit on a shared drive. Edited GeoPackages or Shapefiles on a network share are fragile with more than one writer under either model.

	Points to data	Self-contained
Read shared data	Always a reference	Add links it
Edit shared data	Writes to the source	Edit Source, then Save Edits to Source
A personal analysis result	Written to a file somewhere	Kept in your project
Make a result official	Move it into the shared store	Export it to the shared store
Hand a project to a colleague	Paths must resolve on their machine	One file; linked layers must resolve
Send outside the team	Package	Portable copy

If we choose self-contained: the prototype

A .rgis would become a zip holding a small project.json (maps, styles, layouts, the layer list) and one standard GeoParquet file per stored layer. Layers are stored in the zip uncompressed (Parquet compresses internally), so any layer can be read in place as a byte range from disk, from a browser Blob or over HTTP. RustGIS's GeoParquet reader already works on any byte source.

Tuned GeoParquet (zstd, no bounding-box columns) is about the size of compressed JSON. I checked it losslessly, bit for bit:

Relative to JSON + zstd	Buildings (320k)	Points (320k)	Counties (3,221)
Tuned standard GeoParquet	1.04×	0.77×	0.86×

The gain isn't size. It's that a project wouldn't need to be loaded whole to open, so a 600 MiB project wouldn't need gigabytes of RAM, and saves wouldn't re-encode every layer.

GeoPackage-as-project (see Options considered) would be measured alongside, on the same tests. The prototype has to show:

1. Exact round-trips: values, feature IDs, row order (selections are stored as row positions), empty geometry and aliases.
2. Bounded-memory reopen of the projects above, on the desktop and in the browser.
3. Measured save time and temporary space, on slow disks too. Every save writes a new file.

If we switched:

• opening never rewrites a file;
• the first save of an old project keeps the original;
• old JSON projects open forever;
• the check that tells older builds "made by a newer RustGIS, please update" ships early.

Prototype details: saving, untrusted files, size layouts, effort

Saving. Write a temp file beside the project, copy unchanged layers byte for byte and verify them against the manifest's hashes, then write the changed layers and project.json. Flush and replace atomically: ReplaceFileW on Windows, createWritable in the browser. Appending to the end of a zip can't be made crash-safe. Readers keep the old file open and rebind after a successful replace. If the file changed on disk since it was opened, Save asks rather than overwriting.

Untrusted files. A strict zip profile:

• stored or deflated members only, with no encryption and no data descriptors;
• unique, normalised names, and no overlapping members;
• local headers checked against the central directory;
• bounded reads per member, with caps on counts and sizes;
• ZIP64 allowed.

A smaller, RustGIS-specific layout later. Separate x/y columns, as scaled integers when one decimal precision fits nearly every coordinate, or ALP floats otherwise. That measured 0.44× (buildings), 0.63× (points) and 0.93× (counties) of compressed JSON. The parquet crate writes it without Arrow, so it works in the browser. Two catches: other tools couldn't read those layers directly, and ALP is still a preview encoding in the Parquet spec. Because layer encodings are declared in each file's list of required capabilities, this can be added later without breaking anything.

Effort. Roughly 2,500–4,000 lines across the codec, the in-zip reader, the writer and save, reopening in place, and browser I/O. The browser part is the riskiest.

Groundwork for shared and cloud work

We'll want what established GIS platforms offer: projects opened from cloud storage, offline edits that sync back, and eventually several people editing at once. None of that is proposed here, but a few choices are cheap now and painful to retrofit:

• stable IDs for layers, fields and project items, and persisted feature IDs;
• a list of capabilities each file requires, so older versions refuse what they can't read;
• layer files that can also live in object storage, outside a zip.

These serve all three. The detailed sync design (change records, conflicts, permissions, partial copies) is a separate proposal for when we build one of them. One known bug is worth fixing sooner on its own: offline areas currently overwrite features edited both offline and at the source, without saying so.

Options considered

Keep JSON and compress it: fixes size, not memory

zstd makes projects 3.5–7.4× smaller, and every measured project fits under the limit compressed. But opening still parses everything: the statewide project would need about 2 GB of RAM. Every save still rewrites everything. It's a stopgap at best, and moving to it would mean a second format change soon after.

Reshape the JSON, or a binary serde format: little gain

Compact coordinates, MessagePack, CBOR or WKB in JSON gain at most 11% once compressed. postcard, bitcode and rkyv break old files whenever a field is added. bincode is unmaintained.

The project as a GeoPackage: the strongest alternative

QGIS can store a project inside a GeoPackage, and SQLite makes a strong case as an application file format. You get:

• in-place row edits, transactions and an R-tree;
• data any GIS opens;
• rusqlite already a dependency, with a WebAssembly backend;
• numeric rasters, via the gridded coverage extension.

It loses for now on four counts:

• C SQLite in the core open and save path;
• 2.5–7.5× larger than tuned Parquet for the same layers (counties, buildings, points);
• a live SQLite file in a sync folder or on a network share is risky;
• the browser path is unproven.

If editing very large layers in place becomes the priority, it should win. I'd measure it alongside the zip.

A project file plus a managed data file or folder: halfway between the two models

Saves become proportional to what changed, but it gives up the single file and still needs a package to send. Users would manage two forms of every project.

FlatGeobuf, Arrow IPC, Lance, or a custom format

• FlatGeobuf: an exact spatial index, but uncompressed (about 5× compressed JSON for buildings), and its index at the front means any change rewrites the layer.
• Arrow IPC: good for scratch, but bigger.
• Lance: handles versions well, but it's a multi-file dataset.
• A custom single file: would duplicate what zip plus Parquet already provides.

More thresholds and automatic linking: ruled out

Deciding where a layer lives by count, size or format is what made today's behaviour unpredictable. More rules make it worse.

Questions

1. What is a project: should it point to its data, or be self-contained by default? I lean towards self-contained, as long as the prototype shows it can open large projects without loading them whole. If it can't, pointing to data is the better answer.
2. Browser: must the browser reopen large projects with bounded memory before we switch, or can it follow later? I'm unsure. Holding the desktop back would delay the fix for the people hitting the limit now. But the web app is one of RustGIS's strengths, and letting it lag sets a precedent. Either way, the browser would open and save the new format correctly from day one and be no worse than today.

I'd like to hear from @opengeos/rustgis-maintainers and other contributors before I build anything.

[pick2510]

This is an important question.

Standard GIS behaviour is a project file that links to its data. QGIS and the other desktop GIS work like this, and most users will expect it from RustGIS too.
Today's .rgis is a mix: layers loaded whole and all tool results are stored in the file, while streamed layers and rasters are only linked. Users can't predict which, and that is how the files outgrow the limit.
First of all, backward compatibility is important. Every .rgis saved so far, with its stored layers, has to keep opening.
A lightweight project format also makes sharing easier. A project of a few hundred KB can live in git, in a sync folder or in an e-mail. The data stays where it belongs: in a database, on a shared drive or in a GeoPackage.

So why not both, but as a project and a package?

• The default .rgis links to its data and stays small. Tool results and edited layers are written to a GeoPackage next to the project.
• A separate package file, with its own extension (like .qgz or .ppkx elsewhere), holds the project and its data, for sending or archiving. Opening it gives a normal project again.

The zip with GeoParquet layers proposed here would be a good package format. This means:

• RustGIS has only one project model to maintain, not two equal formats.
• The user always knows which kind of file they have, and whether sending it alone is enough.
• Small project files and instant saves, with no size limit apart from the data itself.

The browser needs some care, since it has no file paths. But I think that is a smaller problem than making every project self-contained.

[deniial00]

Two things from checking today's code that bear on the project + package split:

• .rgispkg already exists, but it isn't yet the package you describe. It embeds only local raster files, base64 inside the project JSON, from a budget shared across the whole package: 512 MiB on desktop, 128 MiB in the browser (crates/engine/src/cmd/items.rs:399-420). Remote COGs, streamed vector files, databases and web layers stay linked, and a raster that no longer fits stays linked without a notice. So a package today can still depend on the original data, and nothing says so.
• "Opening it gives a normal project again" is what happens today: a package opens as a new unsaved project, and Save writes .rgis (docs/user-guide/workspace.md). That works for archiving. For offline work, though, the first Save turns the snapshot back into a referencing project. I'd make that an explicit choice (open decision 10 in my comment).

On tool results in a GeoPackage next to the project: Mavrick measured GeoPackage at 2.5–7.5× the size of tuned GeoParquet for the same layers. Wherever results go, I'd record their provenance (tool, parameters, input references and versions), so a result can be marked "possibly out of date" when its input changes, instead of being recomputed silently or not at all.

[pick2510]

One gap: GeoParquet has no rasters. A package would need them as separate files, Cloud-Optimized GeoTIFFs for example, which can also be read in place from the zip.

[deniial00]

Agreed. Today's package doesn't do this yet: .rgispkg stores embedded rasters base64 inside the project JSON (crates/engine/src/cmd/items.rs:399-420), so nothing in it can be read in place. For COGs inside a zip to keep their range reads, they have to be stored members, not deflated; a COG is compressed internally anyway.

Two more points from my proposal:

• Copying a raster into a package and converting a GeoTIFF to a COG are separate steps. A large GeoTIFF shouldn't be converted silently on Save.
• For packages on S3, rewriting after a small change is shaped by the multipart rules: every part except the last must be at least 5 MiB (AWS limits). Unchanged COGs can be copied on the server side, but changed members would have to go at the end, so the zip layout needs to allow that.

[Mavrick-F]

Thanks @pick2510, this is a good challenge. We agree on a lot:

• Every existing .rgis keeps opening, whatever we choose.
• Rasters belong inside as Cloud Optimized GeoTIFFs, read in place like the layers. That was in the design, and I trimmed it from the post, so thanks for catching it.
• One project model, not two equal formats.

It helps to look at where today's mix came from. In v0.1 a project was one file holding everything. Every exception since, from streaming large files to linking them, was a workaround for memory, because storing data in the project meant loading all of it.

Traditional desktop GIS separates the project from the package for a similar reason: until recently, keeping data in one file meant reading the whole file. GeoParquet and COG change that. A layer stored in the project can be read in place, a byte range at a time, from disk, from a browser Blob or over HTTP. So a project can hold its data and still open without loading it all. I think that's a genuinely new possibility, and a good one to build RustGIS around.

I'd also question whether a small linked project is easier to share. Emailing a bare project works only if the recipient can reach the same data at the same paths. That's how projects broke in the Shapefile era, and why packages exist. A self-contained file is the one you can send without thinking, in the browser too. Shared and authoritative data still stays in its database or on its shared drive, linked as before; only the project's own work lives inside.

Git is a fair point. Most GIS users don't keep projects in git, but the scripting and reproducible-analysis users RustGIS aims at do, and a zip doesn't diff or merge. A diff driver only helps locally: GitHub still shows "binary file changed", and merges don't work. The better answer is the folder form already in the design for cloud storage: the same project.json and layer files in a directory instead of a zip. project.json is then plain text that diffs and merges, and the layer files can go in Git LFS or be ignored.

The costs are real, and they're why the prototype comes first. Every save writes the whole file, and editing a large stored layer is less natural than in a GeoPackage. If the prototype can't open large projects with bounded memory and save them at a reasonable cost, your model is the better answer.

The prototype is the same work under both models, so I'd start there: the codec, in-place reading including COGs, and measured open and save costs. Those numbers tell us whether the zip is the working file or only the package.

[Mavrick-F]

A shorter way to put it: this proposal flips the default rather than inventing a model.

RustGIS is already self-contained by default. Layers loaded whole, tool results, edited layers and browser uploads are all saved inside the .rgis, and that's what users rely on today. What's breaking is the storage underneath. The JSON has to be loaded whole, so large layers were split off as links, and even then the files outgrow the limit. That's the mix we have now.

Both models have the same pieces: a project file, links to shared data, and a self-contained file with the data inside. The difference is which one is the default.

• Traditional: a project links to its data, and you make a package when you need to send it.
• Proposed: a project holds its own work, as RustGIS's projects already do, and you link to shared data when it lives somewhere else.

So the proposal is to fix the model RustGIS already has rather than switch to a new one. It's the same container either way: the zip with project.json, GeoParquet layers and COGs. The question is whether it's what you open every day or only what you send.

[pick2510]

I think @giswqs should give his opinion here. It depends also on the longer term goals. Do we want to develop RustGIS into a very capable GIS platform for big projects or keep the ease of use and lightweight goal.
I tend towards a capable GIS platform replacing most of the tools we are using today. I am working under the hood to remove all the memory and cpu limits from the desktop perspective. But the webversion will probably deviate a lot from the desktop during the development.

[Mavrick-F]

Agreed, this needs @giswqs's view on direction, and I'm with you on the goal: a capable GIS that can replace the tools people use today.

I don't think self-contained by default works against that. Big shared data is linked exactly as in the traditional model, and the desktop reads it in place either way. Your work on removing the memory and CPU limits helps both models; it's also what would make reading layers stored in a project fast. What changes is where the project's own work lives, not how much RustGIS can handle.

It does ask experienced users to relearn some habits. Add links and Import copies, and tool results live in the project unless you write them to a file, which matters when other tools pick them up next. Teams with established procedures would feel that more than individuals.

On the web: its features can lag behind the desktop, but I'd keep one project format, so a project opens in both.

[deniial00]

On "whether it's what you open every day or only what you send": from the remote-data side, the file I open every day is almost all links (COGs on a drive, PostGIS, SQL Server). So for me the question is mostly where my own results go, and how the file behaves once others share it.

If the goal is a capable GIS for teams, two things should be settled before the format, because multi-user work builds on them:

• Change detection on the project file: a version or ETag check before saving, so a save never silently overwrites someone else's. On S3, conditional writes (If-Match) work on CompleteMultipartUpload (AWS).
• Whether a project log is saved in the file. The session log from Session log: every command run in a session, with its time, parameters and outcome (#797, part 1) #947 is memory-only, and it can't be replayed: kept_params (crates/engine/src/project_log.rs:47-55) keeps documents as their size only and clips parameters to 4096 bytes. Merging competing saves needs complete commands, so that would be a second record. Deciding it after the format means retrofitting it.

And agreed on Add links, Import copies. I'd add that every copy keeps its source reference and when it was made, so it can be checked and refreshed per layer later. Copying database data into a file should also warn that the copy can be read without the database's permissions.

[deniial00]

Most of my work is with remote and fairly large data: COGs, GeoTIFFs on network drives, PostGIS and SQL Server. I like the idea of a self-contained project, but in my workflow almost every input stays linked under either model, because nobody copies a 20 GB COG or a production table into a project file. So what I mostly care about is how links, snapshots and handoffs work. I worked through that as a proposal. It sits between the two positions here: @pick2510's split, with @Mavrick-F 's zip as the package. It's a basis for discussion, not a spec. I've marked what I propose and what's still open.

Summary

Two formats that are easy to tell apart: a compact, mostly referencing .rgis project and a largely portable .rgispkg with embedded snapshots. A package keeps the original references, so each layer's snapshot can be refreshed later. COG and GeoParquet are the preferred formats for large rasters and vectors.

The key UX point: you choose the format in the normal Save dialog, not only in a separate export step. RustGIS shows a quick estimated size for both options up front, clearly marked as an estimate.

Deliberately undecided: where derived results physically live in the .rgis model, how completely hard-to-copy services can be packaged, whether a single zip package can be written efficiently on S3, and how far project interoperability should go.

Goals and non-goals

Goals:

• Make it immediately clear whether a project needs external data or opens on its own.
• Don't copy or convert large originals unless needed; keep streaming and range requests.
• Combine portability, provenance and per-layer refresh, without surprise overwrites.
• Keep processing results and their dependencies traceable.
• Try out cloud storage of large packages instead of ruling it out early.
• Respect existing RustGIS features and decisions; prepare for collaboration without overloading this step.

Not in the first step: real-time collaboration (WebSockets, distributed locks, automatic conflict resolution); a guarantee that every GIS source converts losslessly to a neutral file format; a promise that a RustGIS project opens in other GIS-Programs without conversion.

Today versus the proposal

This is a change to what the formats mean, not just a new extension. Both extensions exist already:

	Today (checked at c40daf40)	Proposal
.rgis	Stores layers loaded whole and tool results; streamed layers and rasters as references	A mostly referencing working file; where results go is still open
.rgispkg	Embeds only local raster files, base64 inside the JSON, from a budget shared across the package: 512 MiB on desktop, 128 MiB in the browser (MAX_EMBED = zip::MAX_TOTAL / 2, crates/engine/src/cmd/items.rs:399-420). A raster that no longer fits stays linked without a notice. Remote COGs, streamed vector files, databases and web layers always stay linked. The docs' "Everything, rasters included" (data-formats.md) overstates this.	A usable snapshot of everything that can sensibly be copied, with the original references kept
First save	Save / Save As writes .rgis; packaging is Publish › Package › Project	Both formats in the Save dialog, with estimated sizes
Working on a package	Opens as a new unsaved project; Save writes .rgis (workspace.md)	Save / Save As for packages defined consistently

One project model for both formats

Both formats use the same logical project description. A layer has a stable ID, its style, optionally a reference to its original source and, once materialized, a reference to a snapshot. The container must not change what a layer means.

Building blocks:

• Project manifest: maps, layer tree, styles, layouts, settings and the format version.
• Source reference: provider type, address or ID, dataset or table, filter, CRS, and a stable connection name; never secrets.
• Snapshot descriptor: where it's stored, format, extent or filter, when it was made, the source's version, and check information.
• Derived layer definition: inputs, operation, parameters, the result's reference, and the inputs' versions.
• Source capabilities: can this source be checked for changes quickly, exported in part, or copied completely?

Step	Referencing project (.rgis)	Package (.rgispkg)
1. Load a source	The PostGIS or S3 reference is authoritative	The same original reference is kept
2. Open	Data comes from the original source, with its permissions	The saved snapshot is readable at once, offline too
3. Check	A live source is current anyway	The source can be checked for a newer state, if it supports that
4. Update	Derived results don't update automatically	A layer's snapshot is refreshed only after you confirm

The manifest structure, relative paths, version migration, file naming and transaction boundaries are design questions. This is a model, not a JSON spec.

The nine topics

The proposal in depth, topic by topic

1. Formats and architecture

Proposed: .rgis and .rgispkg stay two visible options on one project model. .rgis is the compact working form with external references. .rgispkg is a package for offline use and handoff, with snapshots embedded as well. "Self-contained" means usable copies of the data, not necessarily the full originals or editable web services. You choose the format on the first save; Save As can convert later. What a plain Save does on an opened package, and whether you keep working in it as a package, has to be defined.

Open: May .rgis embed its own tool results, or must those live outside too? What happens to existing embedded in-memory data when old projects migrate?

2. Sources and snapshots

Proposed: Every copy keeps the reference to where it came from. In a package, the snapshot and the original reference sit side by side. Add means reference, Import means copy on purpose. That avoids accidental copies, especially of large files on network drives and database tables. How that fits the existing import paths needs checking.

Databases allow subsets, SQL filters or spatial extracts, and those must not be mistaken for a full copy. In the code today, though:

• PostgisOrigin and SqlServerOrigin have no filter field (crates/map/src/source.rs).
• A PostGIS subset only works as an SQL layer through DuckDB.
• layer.addSql refuses SQL Server (crates/engine/src/cmd/layer.rs:2006-2010).

So SQL Server subsets would be new work.

For WMS, only rendered images at a given extent and resolution can be copied; that copy isn't equivalent to the service.

Offline areas already do part of this. offline_sync (crates/engine/src/cmd/workspace.rs:641-658) writes edits back with conflict warnings and then refreshes the area. It syncs against the source map in the project, though, not against the database or file behind it.

Open: Which sources or layers may be copied by default? What happens when a dataset is too large or can't be reached? What provenance and filter information does a reliable refresh need?

3. Open data formats

Preference: GeoParquet for copied vectors, COG for larger rasters. GeoPackage stays important for import and export, but not as the general package container.

GeoParquet allows column reads and skipping row groups you don't need, as long as the metadata, writer and reader support it. A COG allows efficient spatial reads through its internal tiles and HTTP byte ranges. Both only keep those properties if the container, compression and reader allow access by file offset.

Converting every dragged-in GeoTIFF to a COG is not decided, and it shouldn't happen silently, especially for large files. "Use the file", "copy it into the package" and "optimize as COG" are three different actions.

Open: Exceptions for small editable vector layers, Zarr, point clouds or special raster products? Keeping the CRS and data types? Should GeoParquet become the default for exported results?

4. Saving, packaging and data ownership

Proposed UX: The normal Save dialog shows both formats with an estimated size, like the format choice in Office (but more elaborate explanation on benefits/disadvantages):

File type in Save dialog	Example display	Meaning
RustGIS project (.rgis)	about 17 KB	Project definition and references; saved results per the open model
RustGIS package (.rgispkg)	about 50 MB	Project definition and embedded snapshots, original references kept

The numbers are illustrative.

• Speed and accuracy: the first estimate should appear within a few seconds. The aim for typical sources is roughly ±20 %, not a guaranteed figure:
  • database statistics can be stale;
  • GeoParquet compression depends heavily on the values;
  • a WMS export has no known size until it's rendered.
• How to estimate: combine known object sizes (S3 HEAD, file metadata), table statistics, samples, and an allowance for container and metadata.
• Weak inputs: when the inputs are weak, show a range ("40–70 MB") instead of false precision.

Company data and permissions: a referenced database stays protected by its access control. A copied snapshot is a standalone copy that anyone with the file can read, regardless of those permissions. RustGIS can't judge whether a copy may be shared, legally or organizationally. So: a clear warning and a deliberate action, and never credentials in the project file. The last part is already the case today: saved addresses drop passwords, keys and tokens.

Open: How are packages labelled when their WMS or service data is only partly reproducible? Is a "complete package" a hard promise, or a snapshot with documented limits?

5. Refresh and sync

Proposed: On opening, show the saved state first. Detect changes in the background where that can be done reliably. Refresh only on request, per layer.

The status distinguishes four cases: change detected, no change after a reliable check, couldn't check, source unreachable. It shows two timestamps: "snapshot made / refreshed on …" and "source last checked on …". Unchecked doesn't mean out of date.

For COGs and objects on S3, ETag, version ID, Last-Modified and size help. For database tables, general PostgreSQL statistics or MAX(updated_at) aren't always enough: deletions, permission limits, triggers, missing change columns or changed filters can go unnoticed. So provider-specific capabilities are the right abstraction, not a forced full-table diff.

The interface shows a subtle hint on each layer, plus a central overview with optional "Check all / Refresh all". A successful check doesn't refresh derived layers.

Open: Check interval, offline behaviour, authentication on refresh, conflicts with locally edited snapshots, and what "automatic refresh" means for live layers versus derived layers.

6. Processing and derived layers

Proposed: Results survive reopening. Parameters and input dependencies are recorded too. An outdated result is marked first, not recomputed.

Example: a 500 m buffer on a PostGIS layer shows as its own layer and can be used tomorrow on the state it was made from. If the input changes later, it shows "possibly out of date". Only a deliberate re-run makes a new state. That keeps traceability and reproducibility separate from automatic freshness.

This doesn't need the full command log in the project. Per derived layer, defined provenance is enough: tool ID, parameters, input references and version indicators. That's different from a chronological session audit trail. Today, tool outputs carry no reloadable source. The closest existing thing is the results package (.rgisgpk, geoprocessing history and outputs).

Open: Where results live in .rgis: a file next to the project, a location the user chooses, a temporary cache, or small results embedded. Also: versioning results, cache invalidation, and incomplete or failed processing.

7. Remote storage and S3

Goal / PoC: Open and read large .rgispkg files directly on S3 without downloading them whole. Look into efficient partial writes explicitly.

A reader can fetch a zip package's central directory with an HTTP range request, then read uncompressed members directly. If a COG or GeoParquet inside the zip is DEFLATE-compressed, efficient random access to its contents is typically lost. So store already-compressed assets "stored" in the zip, and test the implementation for real.

S3 objects can't be overwritten at arbitrary byte positions. Multipart upload with UploadPartCopy can copy unchanged byte ranges into a new object on the server side, and new zip metadata and changed members are then assembled correctly. That can cut client traffic. It doesn't remove costs, latency, rewrites or concurrency issues, and it needs a writer built for it. RustGIS has no S3 write path today, so this is all new work. Two AWS constraints shape it:

• Every part except the last must be 5 MiB to 5 GiB (limits). A small changed project.json can't be a middle part, so changed members go at the end with a new central directory, which leaves dead bytes in the zip. Does the strict zip profile allow that?
• The object only exists once the upload is completed (overview), and If-Match works on CompleteMultipartUpload (conditional writes). An aborted save leaves the old object untouched, so "appending to the end of a zip can't be made crash-safe" plays out differently on S3 than on a local disk.

For active editing, the comparison is a remote folder with the project JSON and separate COG and GeoParquet objects. A slim .rgis with referenced assets is also a sensible alternative, if the single-object approach proves too expensive or complex in practice.

Open: A single zip as an editable cloud file versus a folder layout on S3; CORS and HTTP range support, network interruptions, costs, and performance when saving small changes in large packages.

8. Multi-user work and change detection

Deliberately limited: Real concurrent editing is a later step. What's needed early is detecting that an open project file was changed by someone else, and never overwriting that silently.

On S3, HEAD, ETag, version ID and conditional writes (If-Match) can be the basis for optimistic conflict detection. An app can check at intervals whether the saved project version is still the same; if not, it says so and blocks an uncontrolled overwrite. That's not a real-time event system and not layer locking.

I think we should reconsider persisting the project log in the project file. Today the session log from #947 lives only in memory, and a saved project or package carries none of it. #797 asked for a project log, and #947 shipped it as "part 1", leaving persistence open. Detecting a conflict is only half the job. Once B knows A saved first, B needs to know what A changed, and RustGIS needs a way to combine the two. Because everything in RustGIS is a command, a log of the commands each save added since the shared base version gives both:

• Who changed what, and when, in a shared project, without diffing two project files.
• A path to merging: replay B's commands since the common version on top of A's save. If they don't touch the same layers or items, that's an automatic merge; if they do, it's a conflict with a clear scope. The merge needs the full commands, so the saved log would have to keep them, with secrets kept out and credentials entered again.
• Polling for new entries becomes cheap. In the folder layout or on S3, the log can be its own member or its own objects, appended without rewriting project.json.

What that needs, and why it isn't free:

• Privacy: the log names the files, addresses and connections the user's commands used, which is why reading it takes the file permission. The docs promise today that "sharing a project shares no log". A package for people outside the team should leave the log out or ask, and a project-level setting should turn it on.
• Today's log can't be replayed. It's an audit trail ("it runs nothing again", workspace.md). kept_params (crates/engine/src/project_log.rs:47-55) keeps a map, project or GeoJSON given as text as its size only, clips the rest to 4096 bytes of JSON (LOG_PARAMS_BYTES), and removes secrets. That's right for an audit trail, but an entry can't be re-run as written. A log for merging is a second, different record: complete parameters for the commands that change the project (not reads like opening a file), with secrets left out by design and the user re-entering credentials where a command needs them.
• An author on each entry: today's entries have at, command, params, ok, error and millis, but no user, so "who" needs a new field.
• Replay isn't always deterministic: a tool run on a live PostGIS or S3 source can give a different result later. A merge should replay edits and style changes, and carry tool results over as results, not rerun them.
• Size: complete parameters can carry whole files (data added as bytes) and documents, so those entries should point at the stored layer instead of repeating it. Older entries get compacted into a checkpoint. Today's log keeps at most 10,000 entries (LOG_LIMIT).
• Stable command meanings across versions, so an old entry still replays the same way. That's [Tracking]: Prepare compatibility safeguards for future RustGIS releases #1405 item 2 ("Define stable command/tool semantics").

Later options on top of that: presence, layer locks, event streaming and conflict merge in the interface. Real locks would need extra coordination; S3 alone offers no presence or WebSocket infrastructure.

Open: A minimal conflict dialog when two saves compete. Whether the log lives in .rgis / .rgispkg, next to it, or both. Opt-in or on by default for shared projects.

9. Interoperability with QGIS and ArcGIS

Goal, deliberately limited: Open data formats make data exchange easier; full project interoperability is a separate task.

A GeoParquet or COG inside the package doesn't mean another GIS can open the RustGIS file as a project with its layer tree, styles and layouts. For QGIS, a .qgs / .qgz export is the obvious route. RustGIS already documents importing QGIS projects, but there's no export. For ArcGIS Pro, its own conversion options or a controlled data handoff would need looking into.

The minimum experience: the data in a package is reachable from QGIS without a proprietary extraction step. Ideally, also a project export that lists what it couldn't carry over. Dragging a .rgispkg into QGIS and reading inside the zip directly are hypotheses, not a proven feature.

Open: Is the minimum "data readable" or "project with styles and layout rebuilt"? Which target applications and style subsets should be supported explicitly?

Example workflows

A–D: four workflows to decide against

A. A large company project with PostGIS and a network drive. A project references a PostGIS table and several GeoTIFFs on a shared drive. On Save, .rgis is offered at a few KB; the data stays at its sources, and only authorized users can see the database. To hand it off as .rgispkg, selected tables or extracts and rasters are copied on purpose. The user is told these copies can be read regardless of database permissions.

B. An offline package with a way back to the source. A colleague receives a .rgispkg with frozen layer snapshots. It works without VPN. Back online, RustGIS detects a new object version for an S3 raster layer. She can refresh that layer specifically; a buffer based on it stays as it is and is marked possibly out of date.

C. A single large package in object storage. A 20 GB package sits in an S3 bucket. Opening it reads only the zip index and the COG tiles or Parquet row groups needed. After a style change, we measure whether a server-side byte-range copy plus a minimal upload gives much less client traffic and a short enough save. The folder approach, with the project definition stored separately, is tested alongside.

D. Two open copies of the same remote project. Users A and B open the same version. A saves first. B sees at the next poll, or at the latest before saving, that the remote version changed. B then chooses to reload, save as a copy, or merge. With a persisted project log, B sees which commands A's save added, and B's own commands can be replayed on top of A's version.

Phases

Phase	Scope	Limit / success criterion
0. Spec	Meaning of .rgis / .rgispkg; source/snapshot model; Save dialog; whether the project log is saved in the file; risk cases	The log decision is made before the format settles, so it isn't retrofitted
1. Core	Persistent source references, package snapshots, provenance metadata, estimated size	Project and package open reliably in RustGIS; no silent copies
2. Refresh	Provider capabilities for change checks; manual per-layer refresh; dependent layers marked out of date	The reproducible state stays until a confirmed refresh
3. Cloud PoC	Remote zip range reads, multipart copy, conditional writes, comparison with a folder layout	Measurements instead of a decision from assumptions
4. Extensions	QGIS export, more providers, better estimates, conflict UI	Parts can ship independently
Later	Merging from the project log, presence and locks, collaboration, automatic reprocessing	Its own proposal and consensus

PoC checks

What the cloud PoC should measure

The PoC should show not just that a remote package opens, but whether it's practical under realistic loads and small changes.

• Test data: packages of about 100 MB, 2 GB and 20 GB, each with at least one COG and one GeoParquet plus several small metadata files.
• Reading: latency to first map, bytes actually transferred, number of GET / range requests, behaviour on a slow connection and on interruption.
• Access: does reading from "stored" zip members work without unpacking the whole package? What happens with normally compressed members?
• Writing: change only a style, add a small layer, replace a large snapshot. Measure upload volume, data copied on the server, duration and S3 request costs.
• Concurrency: two writers, an If-Match / ETag failure, recovery after a conflict, and aborting during a multipart upload.
• Comparison: the same tasks with .rgis plus separately referenced S3 assets, or a folder layout.
• Portability: open a package without the original connections; check snapshot paths, metadata and styles; test other GIS readers separately.

Targets like "save in under X seconds" aren't set yet. The PoC should first collect a baseline and limits; only then can we set a firm requirement.

Risks

Risks and possible mitigations

Risk	Effect	Possible mitigation
"The package contains everything" is misleading	A WMS image or a database subset doesn't replace the raw source	Record snapshot type, extent, resolution and limits in the manifest
Loss of data ownership	Recipients read embedded company data without database access	Explicit warning and deliberate packaging; no secrets; follow organizational rules
Unreliable change checks	"Not changed" is claimed wrongly	Provider capabilities, show an unknown status, no full-comparison promises
Zip rewrites on S3	Small changes can rewrite a large object	PoC, multipart copy, "stored" assets, folder fallback
Concurrent writers	Silent data loss and overwritten changes	Optimistic concurrency via ETag / version check before commit
Worse GIS exchange	Data is open but hard to reach inside a zip	Clear structure, documented export, QGIS PoC, possibly a folder view
Complexity in the first version	Too many features block the core workflow	Separate phases; collaboration deliberately later

Open decisions

1. Results in .rgis: external only, partly embedded, or a configurable project store? This is the main difference from Mavrick's model, where tool results stay in the project.
2. Add / Import: when do we really copy, and does Save copy additional data automatically?
3. Package completeness: what counts as an adequate snapshot for WMS/WFS, database subsets and sources that can't be exported?
4. Formats: when GeoParquet/COG, when deliberately GeoPackage, Zarr or the original format?
5. Size estimate: which sources allow a reliable estimate within a few seconds, and when do we show a wide range?
6. Refresh: how does RustGIS handle deletions, new permissions, source errors, filter changes and local edits?
7. Remote package: is editing a single zip on S3 practical enough, or does the folder layout win?
8. Consistency: when are checks triggered, and how does a conditional commit prevent last-writer-wins?
9. Interoperability: layer data alone, or a QGIS project as close as possible?
10. Save behaviour: save an opened .rgispkg back as a package, or keep converting it to a new .rgis project?

And four from comparing this with the thread and the code:

11. Bounding-box columns: the tuned GeoParquet drops "the per-feature bounding-box columns". Doesn't that also lose spatial row-group skipping for large stored layers? If so, what replaces it?
12. Browser: with no file paths, what does a referencing .rgis resolve against? Saved connections only, or browser storage?
13. Compatibility: an older build would need to refuse a zip .rgis or .rgispkg clearly, not report "not a RustGIS project". I think that's [Tracking]: Prepare compatibility safeguards for future RustGIS releases #1405 item 1 and should ship before the format changes.
14. Project log: save a log in the project file, so shared projects can show who changed what and merge competing saves? Session log: every command run in a session, with its time, parameters and outcome (#797, part 1) #947 left this open. Merging needs complete commands, which today's clipped audit log doesn't keep, so this would be a second record. Its costs are privacy (paths and addresses in the log), size, and stable command meanings ([Tracking]: Prepare compatibility safeguards for future RustGIS releases #1405 item 2).

In short: a slim referencing .rgis and a portable .rgispkg with snapshots, on one project model, both keeping the original sources, with the choice and a rough size made in the normal Save dialog. GeoParquet and COG are the preferred copy formats. Source changes are detected where possible, but snapshots and derived results are never refreshed without asking. Large S3 packages, writing included, get tested in a PoC. Saving the project log in the file should be decided with the format, because multi-user work builds on it. Real-time collaboration and full GIS project interoperability stay separate topics.

[Mavrick-F]

Thanks @deniial00, this is an extremely useful response, and checking it against the code caught real problems.

Several parts I'd adopt as they stand: snapshots that keep their source reference, the four refresh states with their two timestamps, provenance on derived layers ("possibly out of date" instead of recomputing silently), and the warning that a copy of database data can be read without the database's permissions.

I'd still make the self-contained file the default, though. Your proposal made me sharpen what that means:

1. A project holds its own data by default. Files you add are copied in, tool results live inside, and opening it never needs anything else.
2. Databases, services and web addresses are linked by default, because copying them silently would escape their permissions or freeze a live source. You can still take a copy on purpose where the source allows it, to edit for your own use or to keep a snapshot. That copy keeps its source and shows your permission warning. So the rule is by kind of source, not by size, and copying is always a deliberate choice.
3. A very large file asks first: "This file is 20 GB. Link to it instead of copying?" An explicit choice when it matters, not a hidden cut-off. Your 20 GB COG on a network drive never gets copied by accident.
4. A project setting, "link added files instead of copying", for teams working from shared drives. It's saved with the project, so a team's shared projects behave the traditional way without everyone changing their own preferences.
5. Your snapshot model is what makes this safe. Every copied file remembers its source and can be checked and refreshed per layer. Copying by default is only acceptable with that in place.
6. Packaging for sending only matters when a project has links. It turns them into snapshots, with your permission warning, and your size estimate belongs there. Most small projects never need it.

That also means .rgispkg can retire. Once a project holds its own data, rasters included, a "package" is just a .rgis whose links have all been turned into snapshots. So packaging becomes an operation ("make portable copy") that writes an ordinary .rgis, and opening any project says plainly if a linked source is missing. Old .rgispkg files would keep opening; RustGIS would just stop writing them. The simplified map package (one map, visible layers only, geometry simplified) could become options on the same step.

Here's why I lean this way. It's more ambitious than a referencing default, but it fits what RustGIS already is. v0.1 saved everything in the project, and RustGIS still copies small files in today. People rely on that, whether they realise it or not. A referencing default would take it away from the users least able to notice: a student who adds a shapefile from Downloads and emails the project to an instructor gets a broken layer, and the fix ("choose .rgispkg") means knowing in advance you'll need it. A professional who wants links sets it once, per project. Projects that just travel also suit what makes RustGIS different and exciting: a small install, a browser version, newcomers, and agents and scripts that pass projects around. The exceptions that piled up came from the old storage forcing data out of the project. With GeoParquet and COG read in place, that reason goes away.

The costs, stated plainly:

• It's the bigger lift. The container, a lossless codec, in-place reading, snapshot refresh and per-browser save paths. A referencing default skips copy-on-add, and refresh is only needed for packages.
• Copying means converting, unless we keep the original bytes. Re-encoding a Shapefile or GeoPackage as GeoParquet has to be lossless, or say exactly what changed. Where a format can already be read in place (GeoParquet, FlatGeobuf, COG), I'd store the original bytes instead. Like your COG point, converting is a separate step, never silent.
• Files grow with their data, and every save writes the whole file. About 0.3 s for 100 MB on an SSD, but 26 s for 200 MB on a cheap USB stick. Big data belongs linked.
• Large files that stream today would be copied unless the prompt or the project setting says otherwise.

On 11, bounding boxes: you were right. I was actually about to post some test results including that. Dropping the per-row bbox column loses row-group skipping in today's reader: a zoomed-in statewide view read 376 MiB instead of 149 MiB. Keeping the column costs 1.5–2× the size on layers with many small features. The replacement is Parquet's native GEOMETRY statistics, one bounding box per row group. A probe showed the parquet crate writes and reads them without Arrow, the writer builds for wasm, and DuckDB and pyarrow read them. They only help if rows are spatially sorted, so stored layers would be Hilbert-sorted into ~4k-row groups, with a logical-order column so selections and drawing order survive. For a ~1 km view of 320k buildings that reads 3.8% of the layer (0.76 MB) instead of 82% (16 MB). Exports keep the bbox column, because other tools use it.

Sizes for the same data (MB, GeoParquet as above, including feature IDs and order)

	320k buildings	320k points	3,221 counties
.rgis today	97.9	27.5	31.5
GeoPackage	97.4	30.6	15.9
File geodatabase	36.6	11.6	6.4
Zipped file geodatabase	17.2	4.5	5.2
GeoParquet as above	20.0	5.4	6.4

So roughly a zipped file geodatabase, while staying readable in place. Large polygons need a byte limit per row group as well as a row limit: the counties fit in one group, so nothing is skipped.

On today's .rgispkg: agreed. It's a zip around the same JSON with local rasters base64'd in, and the silent fallback when a raster doesn't fit breaks the never-silent rule.

A few smaller points:

• Save dialog: with one default, there's no format choice on every save. Your size estimate fits the packaging step, where it answers a real question.
• 14, project log: worth deciding early, but I don't think it blocks the format. With a list of required features and stable IDs, a log can be added later as its own member without breaking older files. Whether merges replay commands or use change records with old and new values belongs with the collaboration design.
• 7, S3: good catch on multipart. Changed members going last, with dead bytes, is one reason I'd use the folder form for cloud editing and keep the zip for local files and sending.
• 12, browser: saved connections for linked layers, and browser storage for the project's own data. This default is also what the browser needs anyway.
• 13, compatibility: agreed. The check that makes older builds say "made by a newer RustGIS" should ship first.

[deniial00]

Thanks for the detailed response! I understand your reasoning, especially the goal of keeping the saving experience simple for users.

However, I still think there's value in keeping .rgis and .rgispkg as two distinct file extensions. For me, the main advantage is that the extension immediately communicates what users can expect:

• .rgis — a lightweight project file that references external data sources.
• .rgispkg — a self-contained project package that includes the data needed to open and use it independently of the original sources.
  I think GeoPackage (.gpkg) is a useful analogy here: users generally understand it as a single file containing the geospatial data they need, rather than just references to datasets stored elsewhere. An .rgispkg could communicate a similar expectation of portability, while obviously being a RustGIS project package rather than a GeoPackage or an attempt to use the same internal format.

That expectation matters when sharing projects. If a user receives an .rgispkg, they should be able to trust that the required data is included. If something cannot be embedded, RustGIS should make that explicit before saving rather than silently producing an incomplete package.

This wouldn't necessarily require two separate internal project models. Both formats could share the same project metadata structure, with .rgispkg adding the packaged data and any necessary manifest information.

I'm completely open to discussing which format should be the default when saving a project. To me, that's a separate UX decision from whether we should maintain a clear distinction between the two formats.

So I'd still lean toward keeping both extensions, while remaining flexible about the default saving behaviour.

Can you understand my reasoning?

[pick2510]

+1 @deniial00

[Mavrick-F]

@deniial00 I definitely see where you and @pick2510 are coming from. You make good points, and I'm convinced on keeping .rgispkg as a guaranteed complete package. I still lean toward .rgis holding the project's own work by default, with large rasters, databases and services linked, rather than being only a reference file. Partly that's for newcomers, but mainly it's because one file keeps the project and its data consistent: a save commits both together, nothing can drift apart or be left behind when the file is copied or backed up, and .rgis and .rgispkg can share one save path. Either way, we need a proper reference system whichever default wins: named connections rather than paths, copies that remember their source, a clear state for missing or changed sources, and a managed place for results when a project links its data. That's needed in both directions, so we can start on it now.

On the default and what .rgis holds: the tradeoff is real on both sides. A self-contained file travels with the work, saves the project and its data as one unit, and is closer to today's .rgis. A referencing file saves cheaply, keeps shared data where it belongs, and is closer to industry standard. I'd like to prototype whether a large self-contained project can save a small change quickly and safely on slow storage. If it can't, that's a strong case for your model.

Meanwhile, today's format is the worst of both: whether a layer is embedded depends on its format and size, every save rewrites one large JSON file, and packages are silently incomplete. These steps would fix the urgent problems without committing us either way:

1. Safer saves: no item file (map, layer, layout, presentation) is written if RustGIS couldn't open it again. Project saves already refuse in fix(engine): reject projects too large to reopen before saving #1364.
2. Honest packages: say which rasters or sources stayed linked, and stop embedding raster formats that can't be read back.
3. Smaller files: a compressed encoding of today's JSON, about 3.5–7× smaller on the projects I measured. Under the saved-file rules, readers ship a release before writers.
4. Lower memory while saving: stream the JSON to disk instead of building it as one string.
5. Your JSON Schema, with fixtures and a CI drift check, so the current contract is explicit before anything changes.

What these don't fix is memory when opening a large project, because today everything is parsed at once. Solving that needs layer storage that can be read in place, which means choosing where the data lives. That's the more committal decision, and the one the prototype should inform.

[Wolren]

I’d like to add a user’s perspective, as this is close to how I already work.

Implementation details aside, conceptually I'm strongly in favour of keeping both capabilities: a self-contained project that can be sent as one file, and a project that points to data that stays where it is. For small-to-medium projects (up to 1–2 GB of source data in QGIS), the one-file zstd archive is what I have used in practice for a while, as thanks to it there is no folder of sidecars to keep together, and paths don’t break when the project moves to another machine. I’ve implemented this export workflow in a QGIS plugin.

One distinction I’d stress, because I think it changes the design: “self-contained” should not mean one zstd-compressed JSON single compressed blob. The model for archive I use compresses each entry separately and keeps a manifest and seek table, so individual entries can be read without unpacking the whole archive. That maps onto the project-versus-package split raised above. A single compressed blob would keep exactly the open-time and memory costs the proposal measures.

Storing layers as separate, in-place-readable members (Parquet/COG) also leaves room for indexes over the data itself, spatial or attribute indexes per layer, written at save time, instead of only an index of the archive’s contents. It is what I would not want to lose by choosing plain zstd. Per-entry zstd is fine for the package; the working file needs the layers readable in place, otherwise indexes cannot be used without unpacking

On desktop, the number I’d care about most in the prototype is time and peak memory to open a 1–2 GB project, and time to save it again. I wouldn’t hold the browser to the same standard: 1–2 GB is beyond what wasm and browser storage handle comfortably. I’d accept a lower, explicit ceiling there, as long as the new format opens and saves correctly from day one.

[Mavrick-F]

Thanks @Wolren. Dock Export's archive is close to what I have in mind, and agreed on keeping both capabilities. In the current proposal a project holds its own data by default, and databases, services and anything you choose to link stay where they are.

Agreed that one compressed blob would keep the open-time cost, and that's the main reason for storing GeoParquet and COG members uncompressed in the zip. They compress internally by row group or tile, so a reader can fetch part of a layer, not just a whole entry. That's also where your indexes fit.

For your 1–2 GB case, I tested a statewide project: Tennessee buildings and roads, census blocks, full-resolution US counties and the national GNIS file, 1.6 GB of sources.

1.6 GB project	Today's .rgis	Everything loaded	Proposed
Size	646 MiB, won't reopen	—	583 MB
Open + draw at street scale	—	32.5 s, 4.4 GiB	1.7 s, 85 MiB
Open + draw, whole state	—	29.2 s, 4.1 GiB	22.3 s, 1.6 GiB
Save (SSD)	—	—	2.1 s

For reference, QGIS on the same machine took 35 s to draw the 3.2M buildings alone at full extent, using about 330 MB. A self-contained copy there means packaging, which converts the layers.

This is the plain first version, and there's clear room to improve on the same container. Per-row-group bounding boxes and a compact coordinate encoding shrink the files further. Stored overviews make whole-map views quick and light on memory. And saves can skip unchanged layers, so they scale with the edit rather than the project.

Agreed on the browser too: a lower, documented ceiling there, as long as it opens and saves the format correctly.

[deniial00]

I've been thinking about another improvement that could complement either of the proposed approaches to project storage.

Currently, the structure of .rgis files is primarily defined by the Rust types and their Serde serialization. While we already have a project format version, as far as I can tell there isn't a standalone, machine-readable JSON Schema describing the complete serialized project structure.

I think it would make sense to introduce a versioned JSON Schema, maintained directly in the repository, similar in spirit to GeoLibre's approach to its deployment configuration.

This could provide several benefits:

• Documentation: A central, explicit definition of the serialized project structure, including required and optional fields.
• Traceability: Schema changes would be visible in Git history, making it easier to understand when and why the format evolved.
• Backward compatibility: Previous schemas could serve as references when supporting older project files and implementing migrations.
• Automated validation: CI could validate saved project fixtures and detect unintended changes to the serialized format.
• Interoperability: External tools could validate or generate RustGIS project metadata without depending on the RustGIS implementation.

Ideally, the schema could be generated from the Rust types (for example, using schemars), or at least automatically checked against them. A CI check for schema drift would be more reliable than relying on documentation alone.

I'd also suggest documenting a simple rule in AGENTS.md and docs/maintaining.md: Whenever a PR changes the serialized project structure, it must update the corresponding schema and relevant test fixtures.

This would complement the compatibility guidelines proposed in PR #1481. Importantly, updating the schema would not automatically require bumping the project format version: the version should change according to the actual compatibility implications of the change.

I see this as largely independent of the .rgis versus .rgispkg decision. Both could share the same versioned project metadata schema, while the package/container structure and embedded-file rules would be specified separately.

The goal would be to make the RustGIS project format an explicit, version-controlled and testable contract, rather than something defined only implicitly by the implementation.

[elpidioassinaturas]

A user's view from academic work in Brazil: soil and land-cover mapping, teaching, and figures for papers and theses. Almost all my data comes from public web sources: IBGE, MapBiomas COGs, INPE and JAXA, through plugins like BrasilData. Whichever default wins, the converging position here works for me: .rgis holding the project's own work, and .rgispkg as a guaranteed-complete package. Four requirements from this kind of work fit the snapshot model @deniial00 described and @Mavrick-F adopted. I'd like them settled with the format.

1. Licence and credit travel with every copy.

• Today only tile sources carry an attribution (crates/map/src/tiles.rs). A vector or raster layer from a service keeps neither its licence nor the credit it requires.
• Once data is copied into a project or package, that copy is a redistribution under the source's terms. Public data comes with real conditions: MapBiomas is CC BY 4.0, IBGE allows reuse when the source is cited, and JAXA Earth asks for credit to "JAXA" plus the dataset and advance notice for commercial use. Each JAXA collection links its own licence (feat(cubes): JAXA Earth's catalog, read live, in the Data Cubes pane (#1532) #1646).
• So the source reference and every snapshot would record: licence (name and URL), the required credit text, and whether redistribution is allowed.
• Packaging would list them, and say which sources restrict sharing, alongside the database-permission warning already proposed.
• A layout could then fill a "Sources" text from the layers it shows. Brazilian theses (ABNT) and most journals require that line under every map.

2. A manifest that serves as a data citation.

• Journals and repositories (Zenodo, data-availability statements) ask which data, from where, which version, accessed when, and under which licence.
• The snapshot descriptor already holds most of that: source address, version or ETag, when the copy was made, and a checksum.
• If the manifest is documented and machine-readable (the JSON Schema @deniial00 proposed), a package becomes a citable archive of the analysis, and RustGIS can print the data-sources list for a paper.

3. A window of a remote raster as a snapshot.

• I mostly need a municipality or a basin out of national COGs (MapBiomas annual coverage, at 30 m for all of Brazil). Neither copying the whole file nor linking to it fits: copying is tens of GB, and links break when a new collection replaces the URL.
• A snapshot whose descriptor records the source and the window (bbox, overview level) is the right unit. Refresh then re-reads the same window from the source. [Feature]: Read a COG window (layer.addUrl {"bbox"}) off the frame in the app, and from plugins #1330 and feat(plugins): a plugin command's layer.addUrl with "bbox" goes to the app (#1330, part 2) #1581 add exactly these windowed reads for plugins.
• I'd make "copy this extent" a first-class snapshot kind, beside full copies and database filters.

4. Layers made by plugins have provenance too.

• Plugins add layers from services (BrasilData) and run tool chains (a land-cover classification plugin chaining GeoLibre's segmentation and Random Forest tools).
• The derived-layer provenance proposed here (tool, parameters, inputs, versions) would also need the plugin's id, version and command. A plugin result can then be marked "possibly out of date" and re-run through the plugin, which can already re-run its commands ([Feature]: A plugin command that downloads a layer runs again when it lands, so it can go on with it (clip, rename, style) in the web app and, after #1170, on the desktop #1202).
• Classification work also makes many intermediate rasters (probabilities, segments). A per-layer "keep in project / scratch" flag would stop them bloating every save.

None of this changes the container question. Points 1 and 2 are fields in the source and snapshot descriptors, and the JSON Schema. They cost little now and would be hard to add once projects exist without them.

[Mavrick-F]

@elpidioassinaturas thank you, this is the kind of view the thread needed. I agree with all four, and with your point that they're cheap now and hard to add later. Here's where I'd put each one.

1. Licence and credit. The source descriptor would carry the licence (name and URL), the credit text it requires, and whether redistribution is allowed: yes, no, or unknown. A catalogue or plugin that knows the licence fills it in (the JAXA catalogue in #1646 already links one per collection). A file the user adds by hand starts as unknown, and RustGIS says so rather than guessing. A package lists every source's licence and warns about those that restrict sharing, next to the database-permission warning. A "Sources" text in a layout, filled from the layers it shows, follows naturally once the fields exist.

2. A manifest that works as a citation. Agreed. The snapshot descriptor already holds the source address, version or ETag, when the copy was made and a checksum. Documented and covered by @deniial00's JSON Schema, it becomes something a data-availability statement can cite, and printing a data-sources list from it is a small command.

3. A window of a remote raster as a snapshot. Yes: a snapshot kind that records the source, the extent and the resolution or overview level, beside full copies and database filters. Refresh re-reads the same window, building on the windowed reads from #1330 and #1581. If the source address stops answering because a new collection replaced it, the layer shows the source as missing; it doesn't silently read something else.

4. Provenance for plugin layers. The derived-layer record would add the plugin's id, version and command with its parameters, so a result can be marked possibly out of date and run again through the plugin (#1202). A per-layer "keep in project / scratch" flag also fits the way saves would work: scratch layers wouldn't be written into the project, so intermediate rasters don't grow every save.

One more in the same family, going the other way. A copy that remembers its source can also send changes back. Refresh pulls a newer version of the source into the project; publish pushes the project's edits out to the source. That's the everyday case of fixing one parcel's owner in a shared parcels.shp, which today means exporting a copy and swapping files by hand.

• Edits stay in the project until you publish, so they're saved with it and work offline.
• Publish refuses if the source changed since the last refresh or publish, and says what changed.
• It also refuses a rewrite that would lose what the file holds (values a field can't store, Z dropped on read), and otherwise keeps the file's own code page, field widths and .prj.
• The layer shows where it stands: in step with its source, with edits not yet published, with a changed source, or both.

I'd proposed this as a separate editing mode in #1385, with its own record in the project file. As an operation on the source descriptor it needs no separate record, so I'll rescope #1385 to build on this. A file that several people edit live, or one too large to copy, can stay linked, as databases are.

All of this is fields and two operations on one source descriptor, so it doesn't depend on the default. I'll write it into the reference-system design.

[pick2510]

The thread has become long, so here is an overview of the variants proposed so far. Please correct me if I got your position wrong.

Variant	Proposed by	What .rgis holds	What .rgispkg holds	Storage underneath	Tool results and edits	Memory when opening	Cost of a save
0. Today	–	A mix: layers loaded whole and all tool results are stored, streamed files and rasters are linked	Local rasters as base64 in the JSON, within a 512 MiB budget; the rest stays linked without a notice	One uncompressed JSON	Inside the .rgis	Everything at once, 2.4–3.9× the file size	The whole file, on every save and autosave
1. Project + package	@pick2510	Links only, small (git, sync folder, e-mail)	A complete copy: GeoParquet, rasters as COG	Small JSON project; the package is an archive	Written as files beside the project	Small, data is read in place from its source	Cheap
2. Self-contained by default	@Mavrick-F (first post)	All of its own data; databases and services linked	Not needed as a separate format	Zip with GeoParquet and COG members stored uncompressed, read in place by byte range	Inside the .rgis	Low, read in place	Open: can a small change to a large file be saved quickly? (prototype planned)
3. Snapshots	@deniial00	Mostly links, compact	Snapshots that keep their source reference and can be refreshed	Zip; COG and GeoParquet preferred	Managed results location; derived layers keep provenance and are marked "possibly out of date"	Low	Cheap for the project; refreshing the package is an explicit step
4. Compromise	@Mavrick-F (6 Nov)	The project's own work; large rasters, databases and services linked	Guaranteed complete package	Zip as in 2; one save path for both	Inside the .rgis	Low, once read in place	Depends on the prototype in 2
5. Per-entry archive	@Wolren	Both a self-contained and a linking project possible	One-file archive for sending	Each entry compressed separately, manifest and seek table, not one compressed blob	–	Low, only what is needed is read	Moderate
Interim (no format decision)	@Mavrick-F (6 Nov)	As today	Says which sources stayed linked	Compressed JSON (3.5–7× smaller), written to disk as a stream, JSON Schema with a CI drift check	As today	Still loaded whole	Less memory and disk

Additions that fit any variant:

• Reference system (@Mavrick-F): named connections instead of paths, copies that remember their source, a state for missing or changed sources, a managed place for results.
• JSON Schema (@deniial00): versioned, in the repository, with fixtures and a CI drift check.
• Change detection (@deniial00): a version or ETag check before saving, so a save never overwrites someone else's (also on S3).
• Academic use (@elpidioassinaturas): licence and credit on every copy, the manifest as a data citation, "copy this extent" of a remote COG as a snapshot kind, plugin provenance and a scratch flag for intermediate layers.

What is still open is the default: a linking .rgis (1, 3) or one that holds the project's own work (2, 4).

One update on the limit: since #1639, the project size RustGIS opens scales with the machine's memory. The 646 MiB statewide project from the first post now opens on a 16 GB machine. That takes some pressure off, but it doesn't change the memory needed to open a large project.

[pick2510]

To get a view from more users than this thread, I opened a poll on the default: #1660. The options follow the table above. Reasons and comments are better placed here, so they stay with the discussion. @giswqs has the final say.

[Mavrick-F]

I said I'd prototype whether a large self-contained project can save a small change quickly and safely on slow storage. Short answer: it can.

How it works. The project is a zip with one GeoParquet file per layer. A save doesn't rewrite the whole file. It adds only what changed to the end of it, then a new table of contents. Until that table of contents is safely written, the previous save is still there, untouched. The space taken up by old versions is reclaimed now and then by a full rewrite, for example when saving a copy.

How fast. I tested a 583 MB statewide project (3.2 million buildings plus roads, blocks and counties) on a cheap USB stick:

Change	Save time
Change a layer's style	9 ms
Move 10,000 buildings	under 0.15 s
Rewrite the whole file (today's approach)	89 s

For comparison, the same edits in a GeoPackage on the same stick took between 4 seconds and nearly 4 minutes. On an SSD, both approaches were fast.

How safe. I interrupted saves at every step and simulated 3,000 power cuts. Every time, the file reopened as either the previous save or the new one, never a broken mix.

This was a throwaway prototype outside RustGIS, so the next step would be building it properly. The same approach carries over to network drives and to the browser, where projects would be kept in the browser's own storage and saved to disk on request.

So save speed isn't a reason to make .rgis a file that only references its data. The default comes down to what users expect when they save and share projects, which is what @pick2510's poll (#1660) asks.