Build the project database from the file instead of from the drawn folios?

[ispyisail]

Opening a project builds every folio in memory before showing one — about 0.07 s per folio, so 30 s or more for a few hundred folios. The project database (the one behind parts lists, summary tables and the wiring list) is then filled by walking those built folios. That order is what prevents opening a big project without building every folio: the database is the one place that could answer "what is on folio 87?" without building it, but it can only be filled once folio 87 is built.

I checked how much of the database the .qet file alone can give, with a small script that fills the same tables from the file and compares them with QElectroTech's own, on the 24 example projects:

• wires, terminals, symbols, their information and grid positions, folios and title-block fields: identical on every row (3 189 wires, 5 283 terminals, 3 678 symbols, 133 folios);
• symbol labels and wire numbers made from a formula (301 symbols, 872 wires): the file keeps the value from the last time it was worked out, which goes out of date when folios are added or moved (K%total-%id is saved as K1-1 and shown as K3-1). So these are worked out again from the file, by the same code the folios use, and match every time;
• one thing QElectroTech had wrong itself: folio dates a century early, fixed in Fix folio dates a century early in the project database #1134.

So the proposal is a change nobody would see: at open, fill the database from the file for everything above, and from the built folios only for shapes, texts and pictures (text sizes need fonts). It is built as a PR (linked below), with a test that fills it both ways on the examples and requires the same result. Opening does not get faster yet — it only removes the ordering that stops it from getting faster later.

Questions:

• Is this direction welcome, or is building every folio on open meant to stay?
• Working out formulas from the file needs a small change to the autonumbering code: it now reads its variables from a plain structure, which a built folio or the file can fill. Is that acceptable?

PR: #1142

[ispyisail]

Does the database drift out of step with the drawings?

Not today. What changes with #1142:

• When a project opens, the database (parts list, wires, summary tables) is filled by reading the file.
• As soon as anything is edited, it is refilled from the drawn folios, exactly as before.

So it stays correct for the same reason it always has: whenever anything changes, it is refilled from the drawings.

Then why read the file at all? Today it makes no difference. It only matters for a possible later change: opening big projects faster by not drawing every folio at once. Folios not drawn yet cannot be read, so the database would have to come from the file. This PR is the first piece of that.

The catch: once some folios are not drawn, "refill from the drawings" no longer works. Something new would be needed to keep the database and the drawings in step, and that is not designed yet.

In short: nothing new can drift today, because the old safety net still works. The PR is only useful if faster opening is built later, and that would need a new way to stay in step. Is faster opening worth pursuing?

If this is a reply in #1141, the project's writing guide says discussion replies should stay under about 80 words. A shorter version:

Nothing can drift today: #1142 only changes the fill at open. After any edit the database is refilled from the drawn folios, as before. Reading the file matters only if big projects later open without drawing every folio. Then "refill from the drawings" stops working, and keeping the two in step would need something new that isn't designed yet. Is faster opening worth pursuing?