Broken import with Pilosa repo master

[dmibor]

Trying to import using Pilosa cluster master branch - returns error "starting field import for segment: doing import: Server error 415 Unsupported Media Type body:'Unsupported media type" - server expects "application/x-protobuf" but client sends "application/x-binary" to roaring import endpoint

[yuce]

Hi @dmibor. The master of the Go client is not compatible with the master of the Pilosa server at the moment. Once #200 is merged, they will be in sync once again.

[yuce]

That PR was merged, which should fix this problem.

[dmibor]

Sweet! looked like something being not merged, thanks for quick reaction, see roaring imports going through.
Quick question - are below created views normal? I have tons of views now even though my time field setup for month and day views (+ standard) only? and all imports for sure are in one day

standard_4849657524411 standard_4849660268705 standard_4849663016106
standard_484965752441119 standard_484966026870502 standard_484966301610620
standard_4849657527607 standard_4849660271901 standard_4849663019302
standard_484965752760728 standard_484966027190109 standard_484966301930226
standard_4849657530804 standard_4849660275009 standard_4849663022411
standard_484965753080406 standard_484966027500917 standard_484966302241105
standard_4849657533912 standard_4849660278205 standard_4849663025607
standard_484965753391214 standard_484966027820526 standard_484966302560714
standard_4849657537108 standard_4849660281402 standard_4849663028803
standard_484965753710822 standard_484966028140201 standard_484966302880322
standard_4849657540305 standard_4849660284510 standard_4849663035108
standard_484965754030501 standard_484966028451010 standard_484966303510809
standard_4849657543501 standard_4849660287706 standard_4849663041412
standard_484965754350107 standard_484966028770618 standard_484966304141224
standard_4849657546609 standard_4849660290902 standard_4849663047805
standard_484965754660915 standard_484966029090225 standard_484966304780510
standard_4849657549805 standard_4849660294011 standard_4849663054109
standard_484965754980524 standard_484966029401103 standard_484966305410925
standard_4849657553001 standard_4849660297207 standard_4849663066806
standard_484965755300131 standard_484966029720713 standard_484966306680627
standard_4849657556110 standard_4849660300403 standard_4849663070003
standard_484965755611009 standard_484966030040322 standard_484966307000306
standard_4849657559306 standard_4849660303511 standard_4849663073111
standard_484965755930618 standard_484966030351129 standard_484966307311113
standard_4849657562502 standard_4849660306708 standard_4849663082612
standard_484965756250224 standard_484966030670807 standard_484966308261207
standard_4849657565611 standard_4849660309904 standard_4849663085808
standard_484965756561102 standard_484966030990415 standard_484966308580815
standard_4849657568807 standard_4849660313012 standard_4849663089004
standard_484965756880711 standard_484966031301223 standard_484966308900423
standard_4849657572003 standard_4849660316208 standard_4849663092112
standard_484965757200320 standard_484966031620831 standard_484966309211231
standard_4849657575111 standard_4849660319405 standard_4849663101701
standard_484965757511127 standard_484966031940509 standard_484966310170124
standard_4849657578308 standard_4849660322601 standard_4849663114310
standard_484965757830805 standard_484966032260115 standard_484966311431027
standard_4849657581504 standard_4849660325709 standard_4849663120703
standard_484965758150414 standard_484966032570923 standard_484966312070314
standard_4849657584612 standard_4849660328906 standard_4849663161902
standard_484965758461221 standard_484966032890601 standard_484966316190224
standard_4849657587808 standard_4849660332102 standard_4849663171403
standard_484965758780829 standard_484966033210208 standard_484966317140321
standard_4849657591005 standard_4849660335210 standard_4849663174511
standard_484965759100508 standard_484966033521017 standard_484966317451127
standard_4849657594201 standard_4849660338406 standard_4849663212603
standard_484965759420114 standard_484966033840625 standard_484966321260304
standard_4849657597309 standard_4849660341603 standard_4849663215711
standard_484965759730922 standard_484966034160305 standard_484966321571110
standard_4849657600505 standard_4849660344711 standard_4849663241105
standard_484965760050531 standard_484966034471112 standard_484966324110515
standard_4849657603702 standard_4849660347907 standard_4849663295001
standard_484965760370207 standard_484966034790721 standard_484966329500129
standard_4849657606810 standard_4849660351103 standard_4849663304502
standard_484965760681016 standard_484966035110330 standard_484966330450222
standard_4849657610006 standard_4849660354212 standard_4849663352006
standard_484965761000625 standard_484966035421206 standard_484966335200623
standard_4849657613203 standard_4849660357408 standard_4849663386901
standard_484965761320303 standard_484966035740814 standard_484966338690119

[yuce]

We support roaring import for time fields now, so the views are computed at the client side and sent to the server. Those view names are not normal though, they should be like: standard_20180305. I'll dig in more and try to reproduce the issue.

[yuce]

I am able to reproduce this issue if I overflow a timestamp, like:

package main

import (
	"io"
	"log"

	pilosa "github.com/pilosa/go-pilosa"
)

func main() {
	cl := pilosa.DefaultClient()
	schema, err := cl.Schema()
	if err != nil {
		log.Fatal(err)
	}
	ix := schema.Index("ix")
	fx := ix.Field("fx", pilosa.OptFieldTypeTime(pilosa.TimeQuantumMonthDay))
	err = cl.SyncSchema(schema)
	if err != nil {
		log.Fatal(err)
	}

	err = cl.ImportField(fx, newTestIteratorWithTimestamp())
	if err != nil {
		log.Fatal(err)
	}

}

type ArrayRecordIterator struct {
	records   []pilosa.Record
	nextIndex int
}

func NewArrayRecordIterator(records []pilosa.Record) *ArrayRecordIterator {
	return &ArrayRecordIterator{
		records: records,
	}
}

func (ar *ArrayRecordIterator) NextRecord() (pilosa.Record, error) {
	if ar.nextIndex >= len(ar.records) {
		return nil, io.EOF
	}
	rec := ar.records[ar.nextIndex]
	ar.nextIndex += 1
	return rec, nil
}

func newTestIteratorWithTimestamp() *ArrayRecordIterator {
	x := uint64(1 << 63)
	return NewArrayRecordIterator([]pilosa.Record{
		pilosa.Column{RowID: 10, ColumnID: 7, Timestamp: int64(x)},   // overflow
		pilosa.Column{RowID: 10, ColumnID: 5, Timestamp: 1483273800}, // 2017-01-01 12:30:00
		pilosa.Column{RowID: 2, ColumnID: 3, Timestamp: 1520268300},  // 2018-03-05 16:45:00
		pilosa.Column{RowID: 7, ColumnID: 1, Timestamp: 1330965900},  // 2012-03-05 16:45:00
	})
}

Is it possible that some of the timestamps in your data overflows int64 ?

[dmibor]

My timestamp was in nanos - see that you expect it in seconds - fixed that in my code and it started to work ok. Didn't have any problems with this before when timestamps were handled on the server - might be a good idea to duplicate logic from there in the client so that that kind of bugs are avoided?

[yuce]

Thanks for finding out this bug. I've created a separate issue for that: #203

[yuce]

I've pushed a fix to: #204, so timestamps are nanoseconds. Most probably it will be merged today.

[yuce]

The fix is in the master.

[yuce]

@dmibor Can we close this issue?

[dmibor]

yes, works good now

[yuce]

Great!