Novel idea for compressing existing Plots (Linux with fallocate/collapse_range)

[bladeuserpi]

For my PB farm I would prefer to compress my existing plots and would like to present this novel idea, which would save time and energy.

1. Prerequisite: Plots stored on Linux using a filesystem capable of fallocate/ collapse_range, e.g. ext4 and XFS, perhaps others too. Google for "man fallocate"

2. Theory of operation
   a) Read Table 1 in RAM (14GB)
   b) Now read Table2 and overwrite it in place using the (compressed) bits from Table 1
   -> Read 14GB and Write 14GB chunkwise
   c) Call fallocate() so the filesystem gets rid of diskspace for Table 1, the plot file then starts with Table 2
   d) Read and Write parts of C1,C2 and C3 to rearrange absolute file pointers as needed. The read/writes should be marginal
   e) Finished

That would use marginal CPU power and 28GB Read + 14GB Write per Plot

Please let me know what you think about this idea?

[harold-b]

The problem with trying to recompress existing plots is that all of Phase 3 needs to be re-run. It's not simple loading those tables and you're done. You have to decompress (speaking of FSE compression here) all of the tables, and re-compute all of them the same way you would do Phase 3.

This makes it a very unattractive task as there's not much a gain to maintain a new partial-plotter codebase or modifying an existing one just for this, which is not a big win.

I think time is best spent in a tool that makes it extremely easy for the user to replot their existing plots.

[bladeuserpi]

Thanks for commenting.
From existing information on chia website I thought it would just store partial Table1 values in Table2 (and brute-force the missing bits when farming). If my guessing is not correct, I will need GPU and replot all.

[bladeuserpi]

This is from chia.net:

" The new compression method, which we are calling “bit dropping”, for levels 1-7 removes the entire table 1, and places a reduced amount of entries from table 1 into table 2 instead of the back pointers. Compression level 1 uses 16 bits, storing only 16 of the 32 bits required for a k=32. This is problematic when you go to fetch an entire proof of space, of course, because half the entries are missing. These now need to be generated on the fly, requiring some compute to look at all possible matches for the missing entries."

@harold-b: Is that description oversimplified or are there other methods than "bit-dropping"?

[harold-b]

Yes you are understanding it correctly -- that partial values from table 1 are stored in table 2 (and so on to table 3 and so forth as you increase the number of bits dropped). It is a little bit (no pun intended) simplified there so that it is not too technical for the general public.

But when they are stored we are still doing the regular FSE compression steps that plots originally have. That is, Phase 3 remains the same. During Phase 3, every table processed propagates forward as input to the next table to be processed all the way to table 7. So as soon as you modify table 1, you'll have to do all of Phase 3 again. This is all part of the canonical plot format (not newly introduced).

[bladeuserpi]

Being not an expert, but I understood farming starts from Table7 and goes backwards with pointers through the other tables until it finds the proof in Table1. Now all tables would stay the same, but Table2 would not contain the backpointers, but (bitdropped) values ? That technique would only work for the last backstep.

[harold-b]

The problem is that whenever you change a table, all tables that follow it must also be changed for the FSE (the compression used in the status quo plots) compression to work. If you were to only modify 2 tables of an existing plot, you'll end up with an invalid plot, and it won't be farmable.

Which is why I mentioned that Phase 3 would have to be re-run completely, which is the portion that does FSE compression.

The way phase 3 works is:

• Process all tables in pairs, starting from the lowest tables (where lowest is determined by its number). Where the lowest table will be called the L (for left) table, and the lowest + 1 will be called the R table (for right).

• For each pair of tables:

• Take the back pointers from table R, and lookup the values pointed by both pointers in each entry from the L table (which in the first table, are the x values, that is what makes up a proof).

• Convert those 2 L values to a single line point. Now we have a line point table.

• Create an index for each entry in the line point table. Now we have an index table.

• Sort both the line point table and the index table on the line points.

• Compress line points into parks and store in the plot

• The index table, which is now sorted on line points, becomes the new L table for the next iteration

• Repeat until all tables are processed

So you see, if we modify the first and second table in an existing plot, we have to re-convert them to line points, re-sort them and re-generate a new index. The sorting order will be different, so the index will point to different locations, and thus we need to use that new index for the next table in the plot, and so on.

[hajes]

why would you compress already compressed?

I have tried to compress plots, and it comes larger than the original plot thanks to compression algo overhead.

[harold-b]

why would you compress already compressed?

https://www.chia.net/2023/01/20/plot-compression-is-here/

[hajes]

but that is not "compression" - they call it "bit dropping". They're basically shifting proofs to RAM/CPU.

The question is whether there is any advantage for old farmers. Not to mention, converting/reploting of thousands of plots.

[bladeuserpi]

See this for more info: https://www.chia.net/2023/01/20/plotting-chias-future/

[harold-b]

but that is not "compression"

How do you think general purpose compression works? You are always doing extra work when decompressing data.

The question is whether there is any advantage for old farmers. Not to mention, converting/reploting of thousands of plots.

You can certainly start a new discussion thread about that, but it is unrelated to the OPs thread.

[hajes]

How do you think general purpose compression works? You are always doing extra work when decompressing data.

by removing empty space, and counting duplicates for example as zip based does ;-)

or psychovisual tricks in video compression

but you may be right, it might be some kind of very lossy compression

[harold-b]

by removing empty space, and counting duplicates for example as zip based does ;-)

Correct, and because such things are applied to remove information, the CPU has to do extra work at the time of decompression instead of copying data directly is my point :)

but you may be right, it might be some kind of very lossy compression

In this case we're not doing lossy compression all proofs are recoverable from the partial information stored