New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

modules/zstd: add BlockDecoder #1214

Closed

lpawelcz wants to merge 26 commits into google:main from antmicro:51873-block-decoder-top

Contributor

lpawelcz commented Nov 30, 2023 •

edited

This PR adds:

Block Header parsing functions
DecoderMux Proc, which collects data from specialized Raw, RLE, and Compressed Block decoders and re-sends them in the correct order
DecoderDemux Proc, which is responsible for parsing Block Header and sending the obtained data to the Raw, RLE or Compressed Block decoders
BlockDecoder Proc, which aggregates the created Decoders together with Muxer and Demuxer. The block is responsible for decoding complete ZSTD Blocks. Currently, it supports RAW and RLE Block types.

NOTE: this is based on #1213 , please ignore commits from that branch when reviewing.

This is part of #1211

lpawelcz force-pushed the 51873-block-decoder-top branch 2 times, most recently from 51e9672 to 141e325 Compare

December 12, 2023 13:02

rw1nkler mentioned this pull request

Add DSLX Zstandard Codec #1211

Open

lpawelcz force-pushed the 51873-block-decoder-top branch 4 times, most recently from 9c99c3d to 611162a Compare

December 19, 2023 11:39

lpawelcz force-pushed the 51873-block-decoder-top branch 2 times, most recently from be431b3 to 5291294 Compare

December 27, 2023 15:49

lpawelcz force-pushed the 51873-block-decoder-top branch 2 times, most recently from b0f7013 to 82d430e Compare

January 4, 2024 14:28

lpawelcz mentioned this pull request

Channels should be encapsulated by procs in the IR #869

Open

lpawelcz force-pushed the 51873-block-decoder-top branch 3 times, most recently from 8ef2e9c to 3b1da80 Compare

January 15, 2024 14:54

rw1nkler mentioned this pull request

modules/zstd: Add Sequence Executor #1295

Closed

lpawelcz force-pushed the 51873-block-decoder-top branch from 3b1da80 to 07ecf8f Compare

February 15, 2024 11:45

rw1nkler and others added 13 commits

February 21, 2024 14:57


          modules/zstd: Add buffer library

a8c0510

This commit adds a DSLX Buffer library that provides the Buffer struct,
and helper functions that can be used to operate on it. The Buffer
is meant to be a storage for data coming from the channel. It acts like
a FIFO, allowing data of any length to be put in or popped out of it.
Provided DSLX tests verify the correct behaviour of the library.

Internal-tag: [#50221]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add Buffer use-case example

This commit adds a simple test that shows, how one can use the Buffer
struct inside a Proc.

Internal-tag: [#50221]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd/buffer: Add benchmarking rules

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd: Add library for parsing magic number

fc57626

This commit adds the library with functions for parsing a magic number and
tests that verify its correctness.

Internal-tag: [#50221]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add library for parsing frame header

1d11225

This commit adds the library with functions for parsing a frame header.
The provided tests verify the correcness of the library.

Internal-tag: [#49967]
Co-authored-by: Roman Dobrodii <rdobrodii@antmicro.com>
Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd/frame_header: Add benchmarking rules

374b000

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          dependency_support/libzstd: Make zstd_errors.h public

4f0a064

Required for expected_status inference in C++ tests for ZSTD decoder
components

Internal-tag: [#53465]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          dependency_support: Add decodecorpus binary

6ddea32

Internal-tag: [#50967]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add data generator library

b4fb1ec

This commit adds a binary that calls decoding to generate data and loads
it into a vector of bytes.

Internal-tag: [#50967]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add zstd frame header tests

e25cbe1

Internal-tag: [#50967]
Co-authored-by: Pawel Czarnecki <pczarnecki@antmicro.com>
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd: Add common zstd definitions

dd34d02

Internal-tag: [#51343]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add raw block decoder

e6a0e0f

Internal-tag: [#51343]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd/raw_block_decoder: Add benchmarking rules

09267ae

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

lpawelcz and others added 13 commits

February 21, 2024 14:57


          modules/zstd: Add rle block decoder

d31c6eb

Adds RleBlockDecoder responsible for decoding Blocks
of RLE_Block Block_Type as specified in RFC 8878, paragraph 3.1.1.2.2.
https://datatracker.ietf.org/doc/html/rfc8878#section-3.1.1.2.2

RleBlockDecoder communicates through BlockDataPacket channels.
It reuses existing RunLengthDecoder block which is interfaced through
two seprate procs:

 * RleDataPacker
 * BatchPacker

Which are responsible for converting input data into format accepted by
RLE decoder and for gathering RLE decoder output symbols into batches
which are then send out through BlockDataPacket.

Internal-tag: [#51473]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd/rle_block_dec: Specify fifo depths for internal channels

4c19f10

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd/rle_block_decoder: Add benchmarking rules

5689ed9

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd: Add block header parsing library

06616b6

Internal-tag: [#51343]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd: Add SequenceExecutorPacket to common definitions

a91e03d

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd: Add block data muxer library

75bf542

This commit adds DecoderMux Proc, which collects data from specialized
Raw, RLE, and Compressed Block decoders and re-sends them in the correct order.

Internal-tag: [#51343]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>


          modules/zstd/dec_mux: Add benchmarking rules

9ae404f

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd: Add block demuxer library

This DSLX proc responsibility is to dispatch encoded blocks to a correct
decoder: RAW, RLE, COMPRESSED.

It tracks and assigns block IDs.
The ID counter is reset on the frame's last block on the last data packet.

Internal-tag: [#51736]
Co-authored-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Maciej Dudek <mdudek@antmicro.com>


          modules/zstd/dec_demux: Specify fifo depths for internal channels

26bd29d

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd/dec_demux: Add benchmarking rules

50b730b

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd: Add block decoder module

4b40857

This adds a decoder of block data. It decodes block header and
demuxes remaining input data into one of specific block decoders
depending on the type of the parsed block. Then it muxes outputs
from those decoders into single output channel.

Internal-tag: [#51873]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd/block_dec: Specify fifo depths for internal channels

a022286

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>


          modules/zstd/block_dec: Add benchmarking rules

caf2b7c

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

lpawelcz force-pushed the 51873-block-decoder-top branch from 07ecf8f to caf2b7c Compare

February 21, 2024 14:07

lpawelcz pushed a commit to antmicro/xls that referenced this pull request


          [TEMP] Add Block Decoder

a20d74e

google#1214

modules/zstd: Add block header parsing library

Internal-tag: [#51343]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>

modules/zstd: Add SequenceExecutorPacket to common definitions

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd: Add block data muxer library

This commit adds DecoderMux Proc, which collects data from specialized
Raw, RLE, and Compressed Block decoders and re-sends them in the correct order.

Internal-tag: [#51343]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>

modules/zstd/dec_mux: Add benchmarking rules

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd: Add block demuxer library

This DSLX proc responsibility is to dispatch encoded blocks to a correct
decoder: RAW, RLE, COMPRESSED.

It tracks and assigns block IDs.
The ID counter is reset on the frame's last block on the last data packet.

Internal-tag: [#51736]
Co-authored-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Maciej Dudek <mdudek@antmicro.com>

modules/zstd/dec_demux: Specify fifo depths for internal channels

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd/dec_demux: Add benchmarking rules

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd: Add block decoder module

This adds a decoder of block data. It decodes block header and
demuxes remaining input data into one of specific block decoders
depending on the type of the parsed block. Then it muxes outputs
from those decoders into single output channel.

Internal-tag: [#51873]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd/block_dec: Specify fifo depths for internal channels

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd/block_dec: Add benchmarking rules

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

proppy self-assigned this

lpawelcz pushed a commit to antmicro/xls that referenced this pull request


          [TEMP] Add Block Decoder

dcc60c9

google#1214

modules/zstd: Add block header parsing library

Internal-tag: [#51343]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>

modules/zstd: Add SequenceExecutorPacket to common definitions

Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd: Add block data muxer library

This commit adds DecoderMux Proc, which collects data from specialized
Raw, RLE, and Compressed Block decoders and re-sends them in the correct order.

Internal-tag: [#51343]
Signed-off-by: Robert Winkler <rwinkler@antmicro.com>

modules/zstd/dec_mux: Add benchmarking rules

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd: Add block demuxer library

This DSLX proc responsibility is to dispatch encoded blocks to a correct
decoder: RAW, RLE, COMPRESSED.

It tracks and assigns block IDs.
The ID counter is reset on the frame's last block on the last data packet.

Internal-tag: [#51736]
Co-authored-by: Robert Winkler <rwinkler@antmicro.com>
Signed-off-by: Maciej Dudek <mdudek@antmicro.com>

modules/zstd/dec_demux: Specify fifo depths for internal channels

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd/dec_demux: Add benchmarking rules

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd: Add block decoder module

This adds a decoder of block data. It decodes block header and
demuxes remaining input data into one of specific block decoders
depending on the type of the parsed block. Then it muxes outputs
from those decoders into single output channel.

Internal-tag: [#51873]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd/block_dec: Specify fifo depths for internal channels

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

modules/zstd/block_dec: Add benchmarking rules

Internal-tag: [#53329]
Signed-off-by: Pawel Czarnecki <pczarnecki@antmicro.com>

cdleary added the app label

rw1nkler mentioned this pull request

modules/zstd: Add sequence decoding #1362

Draft

Member

proppy commented Mar 29, 2024

should we close this and focus on reviewing #1315 ?

Contributor Author

lpawelcz commented Mar 29, 2024

Closing. The review will take place in #1315

lpawelcz closed this

rw1nkler mentioned this pull request

modules/zstd: Add literals decoding #1430

Draft

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment