Skip to content

@sbinet sbinet released this Nov 24, 2020

groot/riofs: fix overlaping blocks w/ big-file mode

TKeys sizes computation was broken for "big files".
The code was indeed changing the `key.rvers` field to indicate
the "big file" mode was enabled, but it wasn't updating the
on-disk size of the TKey (ie: missing 8bytes).

This CL computes the correct size in one-go instead of fixing it
after the facts.

Fixes go-hep/hep#821.
Assets 2

@sbinet sbinet released this Nov 5, 2020

use new gonum/plot@v0.8.1 and its fix for fonts

Assets 2

@sbinet sbinet released this Nov 4, 2020

update for new ccxrootd test server.

Assets 2

@sbinet sbinet released this Oct 19, 2020

DOI

Release v0.28.0 is out of the oven.

This release contains a major new groot related feature: the ability to write non-split user types, by way of TBranchElement and TLeafElement.

csvutil

groot

  • add support for writing large files (i.e.: with size > 2Gb)
  • changed the way groot streamers are created and handled (read/write) from ROOT ones
  • removed rtree.Scanner and rtree.TreeScanner types, in favor of rtree.Reader
  • add support for reading std::bitset<N>
  • add initial support for reading std::map<K,V>
  • introduce rtree/rfunc.Formula to easily create and use "user-based" formulae (thanks Romain Madar (rmadar) for the improvements!)

hplot

  • use gonum/plot/vg's Gio backend for pawgo
  • various fixes and improvements to hplot plotters (thanks Romain Madar (rmadar)!)

AOB

That's all for today.
Next cycle will probably see some work on the structured tree writing (mostly consolidation and adding missing features) and perhaps some performance improvements on the writing side of things.

ChangeLog

  • 653c8f36 all: update Go-mod
  • fe7367f2 groot/rtree: test any, arr-of(any), vec(any) and vec(vec(any))
  • 0db30afb groot/rtree: test c-var-len-arrays
  • de9193a3 groot/rtree: implement writing of structured-event ntuples
  • 33c9bd26 groot/rdict: streamline StreamerOf, add support for slices
  • 2e69fce9 groot/rtree: add support for no-split to WriteVarsFromStruct
  • 43b8c835 groot/rmeta: add stdint to CxxBuiltins
  • 60cf05d2 groot/rdict: add StreamerLoop visitor
  • 2bfd292b groot/{cmd/root-ls,rcmd}: refactor root-ls as a rcmd.List
  • c5e9a68b groot/{,rarrow,rcmd,riofs,rtree}: properly handle slices of n-dim arrays
  • ee067e76 groot/{rarrow,rtree}: reduce Leaf API
  • cdbafa1f groot/{,rarrow,rtree}: better leaf-kind/shape/type handling
  • ae4a840d groot/rtree: simplify Tree/Branch/Leaf interfaces
  • 16ed6fb7 cmd,groot: remove rtree.{Tree,}Scanner
  • dba77171 groot/rsql/rsqldrv: use rtree.Reader in lieu of Scanner
  • d6ee5e87 groot/rsrv: use rtree.Reader in lieu of Scanner
  • eed0f08c groot/rtree: use Reader in lieu of Scanner
  • a3988c10 groot/cmd/root-gen-datareader: use Reader in lieu of Scanner
  • 1c0b336c groot/rcmd: use rtree.Reader in lieu of rtree.Scanner
  • cf246da1 groot/rtree: fix rchain entry offsets computation
  • 4e85dd24 groot/rarrow: use rtree.Reader instead of rtree.Scanner
  • 6a063617 cmd/root2npy: handle ndim arrays
  • 0be35e71 groot/{rarrow,rcmd,rdict,riofs}: properly handle ndim-arrays
  • 4d49a780 groot/rtree: use flattenArrayType in rleaf
  • 30b9ba32 groot/rtree: streamline use of rtree.ReadVarsFromStruct in scanners
  • c0d6fad2 groot: introduce new x-flat-tree file, testing more ROOT builtins
  • 20003c6f groot/rdict: std-bitset cosmetics
  • 06d30c23 groot/{rcmd,rdict,riofs,rtree}: handle map<K,V>
  • bd316477 groot/rdict: fix std::string streamer usage, add preliminary support for std::map-member-wise streamer
  • 51ce0f59 groot/{rdict,riofs}: add support for std::map<K,V> with ROOT-Cling dict
  • 2381e6ae groot/{rbytes,rcmd,rdict,riofs,rtree}: add support for bitset
  • 400e2b6f groot/internal/rtests: add support for rootcling dictionary generation
  • 7536fc78 groot/{rcmd,rtree}: handle Geant4 recovery baskets
  • f4d329eb groot/rdict: add initial test for ObjectFrom
  • 43282104 groot/rdict: impl r/w rmeta.AnyP, rmeta.Anyp
  • cabbd2cc groot/rdict: test r/w-streamer elem round-trip
  • 012823d7 groot/rdict: streamline rmeta.ULong64+rmeta.Long64 handling
  • e675e8f2 groot/rdict: support finding counters across multi-hop inheritance hierarchies
  • 664ec75b groot/rdict: streamline type generation, support n-dim arrays
  • 631f18d4 groot/{rdict,cmd/root-gen-streamer}: handle array dims via StreamerElement.ArrayDims
  • edce8855 groot/{rbytes,rdict}: add ArrayDims() []int to StreamerElement
  • d5b19154 groot/{rcmd,rdict,rtree}: introduce R/W-streamers
  • 19855690 groot/rbytes: extend StreamerInfo interface, introduce Binder and Counter interfaces
  • f50205ac groot/{rbytes,rdict}: add Encoder/Decoder interfaces for ObjectWise/MemberWise kinds
  • ee55be2b groot/{rbytes,rdict}: add BuildStreamers API
  • 8bb70b75 groot/rtree: split r/w test of []intXX (extract and skip []int8)
  • 711290aa groot/rtree: workaround for std::vector
  • b6db3d51 groot/rdict: add test for parsing var-len D32 array
  • 3e8885f3 groot/{rdict,rvers}: add streamer-info for THashTable
  • c23f8cc9 groot/rcont: implement TRefTable
  • a882ddeb groot/rbytes: export r/w std::vectorstd::string
  • 4672fa25 groot/rmeta: add TypeName2Enum
  • a4fbb9e4 groot/rmeta: add STLstdstring to ESTLType enum
  • 1dd68ee6 groot: add C++ streamer for TString
  • 78f2a495 groot/rbytes: remove test of (internal detail) cap-size in ResizeXYZ
  • b3f3f622 hep: bump github.com/klauspost/compress@v1.10.11
  • b9646023 all: migrate to Gonum-v0.8
  • a39f822a groot: bump to ROOT-6.22/02
  • 5c4a32ea groot/riofs: add support for writing big files
  • 79d17b1e ci: bump to Go-1.15.x
  • 73346da2 xroot: update tests for new ccxrootd in2p3 config
  • d55d1b51 groot/{rcont,rhist,riofs}: use rtests macro to emulate C++ rootls
  • 4b4fa012 groot: bump to ROOT-6.22/00
  • 1a9e6f77 hplot: correctly handle negative bin content in log scale
  • 7d2c4375 hep: bump to gokrb5@v8.4.1
  • 3f458e3a groot/riofs: handle files with no embedded streamer for std::string
  • 98270f3f groot/rtree: introduce Reader.Reset
  • abe8a2be groot/{rcmd,rtree}: handle reading of vector<Char_t>
  • 095a9bc6 ci: add apt-get update
  • 0e658332 hep: add official pkg.go.dev badge
  • f4c3901f xrootd/xrdio: implement correct URI parser
  • 63dcd821 ci: bump golangci-lint@v1.28
  • 47aa4362 pawgo: test plot H2D
  • d898ce9f hplot/cmd/iplot,pawgo: handle multiple plot-windows, blocking app.Main
  • e39ab99f all: bump klauspost/compress@v1.10.10
  • 0f5496cf csvutil: add support for writing slices
  • b0b8bd0b groot: bump to go-mmap/mmap@v0.4.0
  • 234a493a xrootd: do not segfault sending data with nil clients
  • aa37c691 hplot/cmd/iplot,pawgo: use vg-gio + Gio
  • ebbd2a74 groot/riofs: use go-mmap@v0.2.0
  • 04360d78 hbook,hplot: introduce hbook.Count, use in hplot.BinnedErrBand
  • 521c323f groot/rdict: handle more range-specs from streamers for D32/F16
  • dc20b078 groot/riofs: use go-mmap/mmap
  • 0f13550f xrootd{,/xrdio}: do not segfault closing nil clients or nil files
  • 1e01356f all: bump to gokrb5@v8.4.0
  • 45f0797a groot/rtree: add support for rmeta.Long64 rstreamer
  • 122ca822 ci: add a timeout=10m for golangci-lint
  • 5616b92d ci: add back golangci-lint
  • 24781f7c hplot: properly handle hstacks with bands and log-y
  • 4cf183c8 groot/internal/rcompress: streamline handling of uncompressible data blocks
  • 364dcc3f hep: update gonum/{gonum,plot}
  • 3050bc0c groot/cmd/root-ls: clip branch name and title at 60 and 50 chars (resp.) for better display
  • 69f0c8d5 groot/{riofs,rtree}: fix and test concurrent writing of trees in different ROOT files
  • ec2e5b95 ci: (temporarily) disable golangci-lint action
  • fcea6526 {cmd/root2arrow,groot/rarrow}: handle leaves with embedded std::vector
  • e2ffbc7c cmd/root2npy: use rtree.Reader
  • 6c0e8ee8 groot/{rtree,rcmd}: add support for recovered baskets
  • 4e246266 ci: make travis ci less verbose
  • 3b702ec4 groot: implement new rfunc-gen default type name convention
  • 384b9ef3 groot/rdict: add generation of streamer info checksum
  • 4d1e5f08 groot/rtree: fix handling of rleaf-element with a "This" StreamerSTL
  • 5e981213 groot/rtree: add formula test for slices
  • 351d7f17 groot: add support for extracting copyright year from code templates
  • c771d0c9 groot/cmd/root-gen-rfunc: first import
  • e1d8871a hplot: fix HStack band display logic
  • a8592066 ci: remove lint scheduled/cron
  • 508d731b groot/rtree: lift WriteVar out of Writer
  • 9448e168 groot/{rcmd,rtree}: make Copy take a Reader instead of Tree
  • 1b6e3cf3 groot/rtree{,/rfunc}: introduce rfunc.Formula protocol
  • 0334332a hep: add pkg.go.dev badge
  • 488e761e groot/{riofs,rtree}: add Join of trees, associated reader and test files
  • 3d790e70 groot/rtree: simplify Tree interface
  • 32ae501f groot/rtree: fix typo in Chain documentation, rename tchain into chain
  • f94677ef ci: extract lint-check into its own workflow
  • 50964db8 ci: fix action/cache restore key syntax
  • 56b78f79 csvutil/csvdriver,hbook/ntup/ntcsv: use ql@v1.1.0 with driver.ConnBeginTx implementation
  • 56834a47 all: fix lint
  • 98694bc7 ci: weave in golangci-lint
  • 96b87a43 csvutil/csvdriver: staticcheck fixes
  • 54e0051a csvutil/csvdriver: implement new database/sql/driver.XyzContext
  • 47234700 groot/rsql/rsqldrv: apply staticcheck fixes
  • 889375c3 groot/rsql/rsqldrv: migrate to new database/sql/driver.XyzContext interfaces
Assets 2

@sbinet sbinet released this May 20, 2020

DOI

Release v0.27.0 is out of the oven.

This release contains a major groot performance improvement.
As can be seen from groot-bench, groot can now read ROOT files (compressed or not) faster than (naive?) C++/ROOT 6.20/04 code (w/ TTree::Branch or TTreeReader):

name                               time/op
ReadCMS/GoHEP/Zlib-8               19.2s ± 1%
ReadCMS/ROOT-TreeBranch/Zlib-8     37.5s ± 1%
ReadCMS/ROOT-TreeReader/Zlib-8     26.1s ± 3%
ReadCMS/ROOT-TreeReaderMT/Zlib-8   25.6s ± 5%  (ROOT::EnableImplicitMT())

This was achieved by:

  • re-engineering the reading code into dedicated rleafXyz, rbasket, ... types
  • introducing a rbasket (concurrent) pre-fetcher

There are still a few low-hanging fruits to reap performances wise (reducing memory usage, reusing rbasket buffers, reusing rbasket buffers across decompression goroutines, etc...)

cmd

  • GoHEP gained a new command, cmd/podio-gen, that can generate Go types according to PODIO descriptions:
$> podio-gen -h
podio-gen generates a complete EDM from a PODIO YAML file definition.

Usage: podio-gen [OPTIONS] edm.yaml

Example:

  $> podio-gen -p myedm -o out.go -r 'edm4hep::->edm_,ExNamespace::->exns_' edm.yaml

Options:
  -o string
    	path to the output file containing the generated code (default "out.go")
  -p string
    	package name for the PODIO generated types (default "podio")
  -r string
    	comma-separated list of rewrite rules (e.g., 'edm4hep::->edm_')

This command should be useful in the connection with Key4HEP.

fit

fit gained the ability to fit multivariate functions, thanks to @JCCPort (Josh Porter):

fmom

fmom gained a couple of functions (under the friendly pressure and w/ contributions from @rmadar (Romain Madar)):

  • fmom.Dot(p1, p2 P4) P4
  • fmom.Boost(p P4, vec r3.Vec) P4
  • fmom.BoostOf(p P4) r3.Vec

groot

groot/rtree gained a new type rtree.Reader, that allows to more easily (and faster) read data from a tree

func ExampleReader() {
	f, err := groot.Open("../testdata/simple.root")
	if err != nil {
		log.Fatalf("could not open ROOT file: %+v", err)
	}
	defer f.Close()

	o, err := f.Get("tree")
	if err != nil {
		log.Fatalf("could not retrieve ROOT tree: %+v", err)
	}
	t := o.(rtree.Tree)

	var (
		v1 int32
		v2 float32
		v3 string

		rvars = []rtree.ReadVar{
			{Name: "one", Value: &v1},
			{Name: "two", Value: &v2},
			{Name: "three", Value: &v3},
		}
	)

	r, err := rtree.NewReader(t, rvars)
	if err != nil {
		log.Fatalf("could not create tree reader: %+v", err)
	}
	defer r.Close()

	err = r.Read(func(ctx rtree.RCtx) error {
		fmt.Printf("evt[%d]: %v, %v, %v\n", ctx.Entry, v1, v2, v3)
		return nil
	})
	if err != nil {
		log.Fatalf("could not process tree: %+v", err)
	}

	// Output:
	// evt[0]: 1, 1.1, uno
	// evt[1]: 2, 2.2, dos
	// evt[2]: 3, 3.3, tres
	// evt[3]: 4, 4.4, quatro
}

root-dump has been updated to use that new way of reading data.

Experimental support for ROOT::RNtuple was added -in a still very WIP fashion- into the groot/exp/rntup package.
When ROOT7 is out (or when ROOT::RNtuple is more stable), this package is expected to graduate to groot/rntup.

The rtree.Reader type has also gained the ability to declare and evaluate user provided functions, taking a list of leaf names and the function to evaluate on the tree data:

func ExampleReader_withFormulaFunc() {
	f, err := groot.Open("../testdata/simple.root")
	if err != nil {
		log.Fatalf("could not open ROOT file: %+v", err)
	}
	defer f.Close()

	o, err := f.Get("tree")
	if err != nil {
		log.Fatalf("could not retrieve ROOT tree: %+v", err)
	}
	t := o.(rtree.Tree)

	var (
		data struct {
			V1 int32   `groot:"one"`
			V2 float32 `groot:"two"`
			V3 string  `groot:"three"`
		}
		rvars = rtree.ReadVarsFromStruct(&data)
	)

	r, err := rtree.NewReader(t, rvars)
	if err != nil {
		log.Fatalf("could not create tree reader: %+v", err)
	}
	defer r.Close()

	f64, err := r.FormulaFunc(
		[]string{"one", "two", "three"},
		func(v1 int32, v2 float32, v3 string) float64 {
			return float64(v2*10) + float64(1000*v1) + float64(100*len(v3))
		},
	)
	if err != nil {
		log.Fatalf("could not create formula: %+v", err)
	}

	fstr, err := r.FormulaFunc(
		[]string{"one", "two", "three"},
		func(v1 int32, v2 float32, v3 string) string {
			return fmt.Sprintf(
				"%q: %v, %q: %v, %q: %v",
				"one", v1, "two", v2, "three", v3,
			)
		},
	)
	if err != nil {
		log.Fatalf("could not create formula: %+v", err)
	}

	f1 := f64.Func().(func() float64)
	f2 := fstr.Func().(func() string)

	err = r.Read(func(ctx rtree.RCtx) error {
		v64 := f1()
		str := f2()
		fmt.Printf("evt[%d]: %v, %v, %v -> %g | %s\n", ctx.Entry, data.V1, data.V2, data.V3, v64, str)
		return nil
	})
	if err != nil {
		log.Fatalf("could not process tree: %+v", err)
	}

	// Output:
	// evt[0]: 1, 1.1, uno -> 1311 | "one": 1, "two": 1.1, "three": uno
	// evt[1]: 2, 2.2, dos -> 2322 | "one": 2, "two": 2.2, "three": dos
	// evt[2]: 3, 3.3, tres -> 3433 | "one": 3, "two": 3.3, "three": tres
	// evt[3]: 4, 4.4, quatro -> 4644 | "one": 4, "two": 4.4, "three": quatro
}

groot also gained a new command: root-split.

$> root-split -h
Usage: root-split [options] file.root

ex:
 $> root-split -o out.root -n 10 ./testdata/chain.flat.1.root

options:
  -n int
    	number of events to split into (default 100)
  -o string
    	path to output ROOT files (default "out.root")
  -t string
    	input tree name to split (default "tree")
  -v	enable verbose mode

root-split allows to split a given tree from a file into n files.

hbook

hbook gained a new "sub-package", hbook/ntup/ntroot that provides convenience functions to expose ROOT trees as hbook ntuples.

H1D can now be subtracted, added, added and scaled (thanks @rmadar)

hbook/yodacnv now supports YODA format version 2 (and its YAML-based metadata description.)

hplot

Many improvement to hplot have been applied thanks again to the interesting suggestions from @rmadar (and his contributions.)
Namely, a new plotter, HStack has been provided:

hstack-example
hstack-band-example

A nice convenience tool to automatically generate (with pdflatex by default) LaTeX PDF plots from the gonum.org/v1/plot/vg/vgtex backend, has been also added to the new hplot concept of a Figure:

Also, a new RatioPlot plotter has been added as well:

ratio-plot

lhef

lhef was improved to be able to handle version 3 of the Les Houches File format.

sliceop/f64s

sliceop/f64s is a new package to apply some "map-reduce"-esque concepts and operations to slices of float64 values.

The Filter, Find, Map and Take functions are available.

AOB

That's all for today.
Writing trees with structured data is still on the roadmark, together with (probably) some performance improvements on the writing side of things.
Also, already in, is a feature similar to TTreeFriends: rtree.Join(trees []rtree.Tree).

Assets 2

@sbinet sbinet released this Apr 15, 2020

groot/rtree: implement a Formula interpreter

Fixes go-hep/hep#634.
Assets 2

@sbinet sbinet released this Mar 27, 2020

groot/{rarrow,rdict,rtree}: add [N] or [X] to generated struct tags i…

…n types generated from StreamerInfos
Assets 2

@sbinet sbinet released this Mar 19, 2020

  • drop golang.org/x/xerrors
  • bump to ROOT-6.20/00
  • introduce cmd/podio-gen a PODIO-like generator
  • add ability to align axes in a hplot.TiledPlot
  • improve compatibility w/ ROOT::TTree::{Draw,Scan}
  • make groot/rtree.Writer.Close() idempotent (so it can be defer'd)
Assets 2