Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RunContainer32 implements a run-length #70

Merged
merged 32 commits into from Dec 17, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
f3fb9e2
runContainer16/32 and the associated
glycerine Nov 24, 2016
caf0e25
bitmaps and arrays know more about rle; supply missing double-dispatc…
glycerine Dec 16, 2016
84b5c27
use bitmaps for unions/or operations
glycerine Dec 16, 2016
5e88adf
use bitmap even for iand
glycerine Dec 16, 2016
2b04cca
restore serialization_littleendian.go and serialization_generic.go
glycerine Dec 16, 2016
f75c987
atg. mild coverage improvements
glycerine Dec 16, 2016
dc554a3
exlucde _gen.go files from coverals
glycerine Dec 16, 2016
e5b4b6d
actually ignore *gen.go files in the .travis.yml
glycerine Dec 16, 2016
4fd3399
atg. add coverage
glycerine Dec 16, 2016
e543557
1. Adding RunOptimize function,
lemire Dec 16, 2016
546544d
Merge pull request #3 from BitmapDevs/master
glycerine Dec 16, 2016
82c817f
Adding a few tests (nothing much)
lemire Dec 16, 2016
8b68d57
atg. fix 051 on data without runs by ignoring the spec and just using…
glycerine Dec 16, 2016
c0333f4
atg. match updated spec for serialization format
glycerine Dec 16, 2016
0f62d28
atg. delete msgpackSerializedSizeInBytes and GetMsgpackSerializedSize…
glycerine Dec 16, 2016
88533af
atg. use newBitmapContainerwithRange as an optimization
glycerine Dec 16, 2016
e7e93ed
Merge pull request #6 from BitmapDevs/master
glycerine Dec 16, 2016
bdeadd3
Minor code review with a few simple changes. Also I trigger a bug with
lemire Dec 16, 2016
65ca9d6
Adding two small questions.
lemire Dec 16, 2016
edff2fb
run containers implemented; users supply own sync
glycerine Dec 16, 2016
9db8196
merge pull request from lemire
glycerine Dec 16, 2016
c56bbde
atg. fix TestExample2_roaring061
glycerine Dec 16, 2016
5baee8f
wip ft. failing tests
glycerine Dec 16, 2016
d2a373e
atg. fix integer overflow problem
glycerine Dec 16, 2016
93bd66c
atg. add test for presence of all 3x3x11 == 99 container inteface imp…
glycerine Dec 16, 2016
22db478
ft. failing test 067 for container agreement
glycerine Dec 17, 2016
f8417a3
ft. 067 but progress
glycerine Dec 17, 2016
6992071
atg. all tests green with 067 consistency checks
glycerine Dec 17, 2016
53a5864
atg. quiet down debug prints
glycerine Dec 17, 2016
5935393
atg. 067 can do fuzzy like multiple passes
glycerine Dec 17, 2016
ac6018a
atg. quiet down.
glycerine Dec 17, 2016
198ccf2
atg. fix overflow issues up near MaxUint16
glycerine Dec 17, 2016
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
12 changes: 6 additions & 6 deletions .travis.yml
@@ -1,11 +1,11 @@
language: go
sudo: false
install:
- go get github.com/smartystreets/goconvey/convey
- go get github.com/willf/bitset
- go get golang.org/x/tools/cmd/cover
- go get github.com/mattn/goveralls
- go get github.com/mschoch/smat
- go get -t github.com/RoaringBitmap/roaring
- go get -t golang.org/x/tools/cmd/cover
- go get -t golang.org/x/tools/cmd/cover
- go get -t github.com/mattn/goveralls
- go get -t github.com/mschoch/smat
notifications:
email: false
go:
Expand All @@ -14,7 +14,7 @@ go:
- tip
script:
- go test -v -covermode=count -coverprofile=coverage.out
- $HOME/gopath/bin/goveralls -coverprofile=coverage.out -service=travis-ci -repotoken KOlKyOXz0qSjAqvfTF28RzNlr3unxjrLh
- $HOME/gopath/bin/goveralls -coverprofile=coverage.out -service=travis-ci -repotoken KOlKyOXz0qSjAqvfTF28RzNlr3unxjrLh -ignore arraycontainer_gen.go,bitmapcontainer_gen.go,rle16_gen.go,rle_gen.go,roaringarray_gen.go
matrix:
allow_failures:
- go: tip
3 changes: 2 additions & 1 deletion CONTRIBUTORS
Expand Up @@ -6,4 +6,5 @@ Elliot Murphy (@statik),
Bob Potter (@bpot),
Tyson Maly (@tvmaly),
Will Glynn (@willglynn),
Brent Pedersen (@brentp)
Brent Pedersen (@brentp),
Jason E. Aten (@glycerine)
20 changes: 19 additions & 1 deletion Makefile
@@ -1,4 +1,4 @@
.PHONY: help all test format fmtcheck vet lint qa deps clean nuke
.PHONY: help all test format fmtcheck vet lint qa deps clean nuke rle backrle ser



Expand Down Expand Up @@ -80,3 +80,21 @@ clean:
nuke:
rm -rf ./target
GOPATH=$(GOPATH) go clean -i ./...

rle:
cp rle.go rle16.go
perl -pi -e 's/32/16/g' rle16.go
cp rle_test.go rle16_test.go
perl -pi -e 's/32/16/g' rle16_test.go

backrle:
cp rle16.go rle.go
perl -pi -e 's/16/32/g' rle.go
perl -pi -e 's/2032/2016/g' rle.go

ser: rle
go generate

cover:
go test -coverprofile=coverage.out
go tool cover -html=coverage.out
36 changes: 19 additions & 17 deletions README.md
Expand Up @@ -28,17 +28,21 @@ http://arxiv.org/abs/1402.6407 This paper used data from http://lemire.me/data/r
- Daniel Lemire, Gregory Ssi-Yan-Kai, Owen Kaser, Consistently faster and smaller compressed bitmaps with Roaring, Software: Practice and Experience (accepted in 2016, to appear) http://arxiv.org/abs/1603.06549



### Dependencies

- go get github.com/smartystreets/goconvey/convey
- go get github.com/willf/bitset
- go get github.com/mschoch/smat
Dependencies are fetched automatically by giving the `-t` flag to `go get`.

they include

- github.com/smartystreets/goconvey/convey
- github.com/willf/bitset
- github.com/mschoch/smat

Note that the smat library requires Go 1.6 or better.

Naturally, you also need to grab the roaring code itself:
- go get github.com/RoaringBitmap/roaring
#### Installation

- go get -t github.com/RoaringBitmap/roaring


### Example
Expand Down Expand Up @@ -130,13 +134,15 @@ bitmaps never use more than 2 bytes per integer. You can call

Current documentation is available at http://godoc.org/github.com/RoaringBitmap/roaring

### Thread-safety
### Goroutine safety

In general, it should generally be considered safe to access
the same bitmaps using different threads as
long as they are not being modified. However, if some of your
bitmaps use copy-on-write, then more care is needed: pass
to your threads a (shallow) copy of your bitmaps.
In general, it should not generally be considered safe to access
the same bitmaps using different goroutines--they are left
unsynchronized for performance. Should you want to access
a Bitmap from more than one goroutine, you should
provide synchronization. Typically this is done by using channels to pass
the *Bitmap around (in Go style; so there is only ever one owner),
or by using `sync.Mutex` to serialize operations on Bitmaps.

### Coverage

Expand Down Expand Up @@ -185,11 +191,7 @@ traces.

### Compatibility with Java RoaringBitmap library

You can read bitmaps in Go (resp. Java) that have been serialized in Java (resp. Go)
with the caveat that the Go library does not yet support run containers. So if you plan
to read bitmaps serialized from Java in Go, you might want to call ``removeRunCompression``
prior to serializing your Java instances. This is a temporary limitation: we plan to
add support for run containers to the Go library.
You can read bitmaps in Go (resp. Java) that have been serialized in Java (resp. Go).

### Alternative in Go

Expand Down