Pipe block operators proposal

Block-template operator

• Excellent parallel
  • map(fn)
  • filter(pred)
• Good parallel (local accumulators)
  • reduce(fn) (makes sure that size >= 1)
  • reduce0(zero, fn) (also takes zero element. Can accept zero-sized stream)
  • group_by(field) (in fact it's reduce by google whitepaper)
  • group_by(fn)
  • partition_by(pred) (??? должен разделять по условию... формат???)
• Bad parallel
  • sort<asc>(field)
  • sort<asc>(field1,field2) Field2 used if field1 was equal
  • sort<asc>(fn)
  • unique(field) (alias disctinct)
  • unique(fn)
• Shrinkable
  • skip(count) -> skip<count>
  • skip(pred)
  • limit(count) -> limit<count>
  • limit(pred)
  • any(pred)
  • all(pred)
• Debug
  • peek(dst)
  • print (==peek(stdout))

Frame operators

• pack<3> N frames into 1.
• unpack 'field' 1 frame into N. (in fact combos with pack-blocking)

Block operators

• Pack each block to make it operational over N-subblock-frame
  • unroll<4>
  • vectorize<4> will vectorize dividable part of unrolled

Alignment

• Array of objects (default)
• Object of arrays
  • Object of mixed arrays
    • Fusing same type into single array
      • pow 2 fusing
    • Partial fusing
    • Rest trash into object
  • How to
    • align<unroll,separate> # Default=object of arrays, may be overriden
      • align_fuse 'field_a', 'field_b' # Force fuse N fields. Transitive. Must be same type in first implementations.
        • align_fuse<4> 'r', 'g', 'b' # Will fill 4th position with 0
      • align_fuse_type 'type_name' # Force fuse for all fields of specified type
      • align_rest 'field_a' # Force field be not unrolled, but putted in "rest" object
    • align<unroll,type_fuse> # Default=object of arrays, all fields of same type fused
    • align<unroll,manual> # Default=all in "rest" object
      • align_unroll 'field' # Remove from rest. Multiple field same as align_fuse. So use 1-liners for separate unroll.
    • align<> # allows autotuner make special alignment for this specific thread (useful for pipe split/join)

Black magic ???

• Loop Stream Detector size/format tuning
  • some info
  • intel info
• CPU cache line tuning
  • frame_size<1MB> # Causes autotuner pack block-operations in blob that will precess 1MB data frame per tick. Round down, but at last 1 (will provide warning if constrain not met).
• CPU time quant granularity rounding/tuning (enough work for thread)
  • time_frame<1ms> # similar to frame_size