Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interface conversion error while live loading exported JSON data #4468

Closed
prashant-shahi opened this issue Dec 24, 2019 · 7 comments · Fixed by #4793
Closed

Interface conversion error while live loading exported JSON data #4468

prashant-shahi opened this issue Dec 24, 2019 · 7 comments · Fixed by #4793

Comments

@prashant-shahi
Copy link
Member

@prashant-shahi prashant-shahi commented Dec 24, 2019

What version of Dgraph are you using?

master

Steps to reproduce the issue (command/config used to run Dgraph).

$ curl 'localhost:8080/admin/export?format=json'
{"code": "Success", "message": "Export completed."}
$ dgraph live -f g01.json.gz -s g01.schema --alpha=localhost:9080 --zero=localhost:5080
...

Running transaction with dgraph endpoint: localhost:9080

Processing schema file "g01.schema"
Processed schema file "g01.schema"

Found 1 data file(s) to process
Processing data file "g01.json.gz"
panic: interface conversion: interface {} is *time.Time, not time.Time

goroutine 45 [running]:
github.com/dgraph-io/dgraph/tok.HourTokenizer.Tokens(0x1818880, 0xc0027920a0, 0x10, 0xc004074020, 0xa, 0xc00328c003, 0xc000131880)
	/home/prashant/go/src/github.com/dgraph-io/dgraph/tok/tok.go:262 +0x3ca
github.com/dgraph-io/dgraph/tok.BuildTokens(0x1818880, 0xc0027920a0, 0x1a230e0, 0x261d298, 0x1a230e0, 0x261d298, 0x0, 0x1, 0xc004074010)
	/home/prashant/go/src/github.com/dgraph-io/dgraph/tok/tok.go:104 +0x63
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).conflictKeysForNQuad(0xc0002e8000, 0xc00348c510, 0x3e8, 0xc004072000, 0xc000045380, 0x0, 0xc00009db00)
	/home/prashant/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/batch.go:301 +0x448
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).conflictKeysForReq(0xc0002e8000, 0xc000459f88, 0xc0007e2001, 0x7d0, 0x800)
	/home/prashant/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/batch.go:322 +0xe9
github.com/dgraph-io/dgraph/dgraph/cmd/live.(*loader).makeRequests(0xc0002e8000)
	/home/prashant/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/batch.go:384 +0x1db
created by github.com/dgraph-io/dgraph/dgraph/cmd/live.setup
	/home/prashant/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/run.go:356 +0x2f0

Expected behaviour and actual result.

This issue is seen only while live loading JSON data and not for RDFs.

Also, I was able to bulk load the exported data in both JSON and RDFs without any problem.

Note: Dataset used here was from JS Flock.

@abhimanyusinghgaur

This comment has been minimized.

Copy link
Member

@abhimanyusinghgaur abhimanyusinghgaur commented Jan 2, 2020

It is happening with RDF dataset as well.
I had tried loading the 1million.rdf.gz in tutorial with following command:
./dgraph live -f ~/dgraph/data/1million.rdf.gz -s ~/dgraph/data/1million.schema

@JimWen

This comment has been minimized.

Copy link

@JimWen JimWen commented Feb 11, 2020

It is happening with RDF dataset as well with the shell below:

nohup ./bin/dgraph live -f ./data --format rdf -s ./schema.txt -a "172.21.46.35:9080,172.21.46.39:9080,172.21.46.40:9080" > live.log 2>&1 &

@MichelDiz

This comment has been minimized.

Copy link
Member

@MichelDiz MichelDiz commented Feb 13, 2020

@sleto-it quick note about this issue. it is not related to #3610

After reading this https://discuss.dgraph.io/t/panic-when-importing-a-bigger-dataset-in-tour/5969 I noticed the "YearTokenizer" error msg. So this is related to Datetime Tokenizers.

If I removed the indexing from the schema it loads normally.

To test just create an empty RDF file(no need to gzip it) and add:

<15113317135919419286> <initial_release_date> "2001-01-01T00:00:00Z" .

and load it:

dgraph live -f 1million.rdf --alpha localhost:9080 --zero localhost:5080 -c 1

if you add @index(year) it will fail.

initial_release_date: dateTime @index(year) .

So, the workaround, for now, is to remove the indexing just in the moment of the load. And then you can add again the indexing that will be fine.

Also, we need to make sure that we have small test cases for each case of a load(or even just for Tokenizers). Thus making sure that won't happen again.

The test was done with RDF in this version bellow

Dgraph version   : v2.0.0-beta1
Dgraph SHA-256   : 178663a98a3d59879a3d5c42928c89eb5f83afc2bfc0093272941e7a53515847
Commit SHA-1     : 6fac5d7c4
Commit timestamp : 2020-01-30 14:45:54 +1100
Branch           : HEAD
Go version       : go1.13.7
@LoganLee1991

This comment has been minimized.

Copy link

@LoganLee1991 LoganLee1991 commented Feb 15, 2020

Something more about this: the @index(float) and @index(int) have the same problem

@iuryfukuda

This comment has been minimized.

Copy link

@iuryfukuda iuryfukuda commented Feb 17, 2020

For fix the load in tour https://tour.dgraph.io/moredata/1/:

Change this:

# Define Directives and index

director.film: [uid] @reverse .
genre: [uid] @reverse .
initial_release_date: dateTime @index(year) .
name: string @index(exact, term) @lang .

To this:

# Define Directives and index

director.film: [uid] @reverse .
genre: [uid] @reverse .
initial_release_date: dateTime year .
name: string @index(exact, term) @lang .

press Run. wait for Load the data and after finish change back again and press Run again with initial config (press Reset and after reset press Run)

@MichelDiz

This comment has been minimized.

Copy link
Member

@MichelDiz MichelDiz commented Feb 18, 2020

@iuryfukuda we can't edit the tour for that. We have to fix the issue. But thanks.

@iuryfukuda

This comment has been minimized.

Copy link

@iuryfukuda iuryfukuda commented Feb 18, 2020

@MichelDiz Yeah, no problem, I only putted a resolution description of the tour in this issue if anyone needs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

8 participants
You can’t perform that action at this time.