Project Zipline

Introduction

Project Zipline is a program to accelerate innovation in lossless compression. With this release Microsoft is making available:

• A new compression format called XP10 which is tailored for modern cloud datasets.
• All specifications for the entire pipeline.
• All RTL to support implementation for the entire pipeline.
• A testbench environment specifically developed for the VCS simulator.

Project Zipline is open-sourced under the MIT License, see the LICENSE file.

Specifications

The following Project Zipline specifications have been included under the “specs” directory in this repository:

• Project_Zipline_Compression_Specification
• Project_Zipline_Huffman_Encoder_Micro_Architecture_Specification
• Project_Zipline_Comp_lz77_Micro_Architecture_Specification
• Project_Zipline_Crypto_Engine_Micro_Architecture_Specification
• Project_Zipline_Decompression_Top_Micro_Architecture_Specification
• Project_Zipline_Keyblob_Micro_Architecture_Specification
• Project_Zipline_KME_Micro_Architecture_Specification
• Project_Zipline_Prefix_Attach_Micro_Architecture_Specification
• Project_Zipline_Prefix_Micro_Architecture_Specification
• Project_Zipline_SSB_Micro_Architecture_Specification
• Project_Zipline_Top_Micro_Architecture_Specification

Simulation Notes

Edit the file ./zipline.setup for VCS, SYNTH, and VERDI environment settings.

Then source the file as follows:

source zipline.setup

Go to either the CCE_64, CDD_64, or KME run directory:

cd dv/CCE_64/run

cd dv/CDD_64/run

cd dv/KME/run

To build the simv executable:

make build_simv

Example for simulating a test:

make run_simv TESTNAME=xp10

Example for simulating a test with waves (.vpd):

make run_simv TESTNAME=xp10 WAVES=1

Example for simulating a test with Verdi waves (.fsdb):

make run_simv TESTNAME=xp10 VERDI_WAVES=1

All of the tests for the CCE_64, CDD_64, and KME can be run via a regress script in the associated "run" directory:

dv/CCE_64/run/regress

dv/CDD_64/run/regress

dv/KME/run/regress

All test files are located in the associated engine directory "tests". Each directory also contains a README file with a description of each test.

dv/CCE_64/tests

dv/CDD_64/tests

dv/KME/tests

A description of the programmable registers in the CCE/CDD/KME can be found in the register_doc directory. Please read register_doc/README for more details.

KME Notes

The KME RTL (rtl/cr_kme) has been modified to remove the following modules:

• AES engine in the random GUID generator
• SHA engines within the KDF function
• AES engine within the Key Decryption logic

With these reductions, the RTL only supports Key Types 1-6 without KDF and Key Type 0 without encryption/authentication

Additionally, the KME output has been reduced to support a single engine.

All modifications are identified with "KME_MODIFICATION_NOTE" comments in the RTL code.

Synthesis Notes

Note: For synthesis, please edit syn/Makefile to choose an LSF or dedicated machine with at least 16G of memory. Edit the contents of this line, before "dc_shell":

bsub -Is -q irv-cpx-M16 -R "rusage [mem=16000]" dc_shell -f syn_example.tcl | tee ./LOGS/syn_eample.tcl.date '+%m%d.%H:%M'.log

To run synthesis for CDD

cd syn/cr_cddip

make syn

To run synthesis for CCE

cd syn/cr_cceip_64

make syn

To run synthesis for KME

cd syn/cr_kme

make syn