Note: This repository is no longer maintained. For our new tool in generating cars, go to delta-car-gen

Δ Delta Chunker

Delta chunker is currently under heavy development

This repo contains the code for the delta-chunker binary. This binary is used to chunk files into smaller pieces (CAR files) for use with Delta deal making engine.

Features

• Tracks all the files in a directory and chunks them into CAR files
• All CAR files can be streamed directly to a live delta instance
• Supports e2e and import deals
• Supports split-size to chunk files into smaller pieces
• Supports delta-url to stream CAR files to a live delta instance
• Supports delta-api-key to authenticate with a live delta instance
• Supports delta-wallet to specify the wallet to use for the deal. Pass the hexed wallet address from boostd/lotus export.
• Supports type to specify the type of deal to make
• For e2e type, it'll prepare the car and send the file over to the delta instance.
• For import type, it'll prepare the deal. You'll need to manually send the car file to the miner. This is something we will automate soon with delta importer
• API endpoint to allow external clients to trigger the chunker to run with a give large blob [WIP]

Build

make all
./dc car-chunk-runner --help

Run chunker task

Prepare the chunk task file

This will be the basis of the car-chunk-runner for making storage deals.

label: "sample-run"
chunk-tasks:
  - name: "chunk-task1"
    source: "./example/source"
    output-dir: "./example/output_dir"
    split-size: "0"
    connection-mode: "e2e"
    include-commp: true
    auto-retry: true
    miner: "f0137168"
    delta-url: "https://delta.estuary.tech"
    delta-api-key: "[API_KEY]"
    delta-wallet: "f1z5ykhx7qi2jeukwlve3mu2zbcuzrtewhc3iajmi" # hexed wallet address from boostd / lotus export

Run the chunker

./dc car-chunk-runner --run-config=<run-config-file>

This will go thru each of the chunk-split tasks and run the delta chunker and make deals with the delta instance.

Author

Protocol Labs Outercore Engineering.