New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dgraph 1.0.12 rc5 crashes with panic #2980

Closed
makitka2007 opened this Issue Feb 6, 2019 · 7 comments

Comments

Projects
None yet
3 participants
@makitka2007
Copy link

makitka2007 commented Feb 6, 2019

If you suspect this could be a bug, follow the template.

  • What version of Dgraph are you using?
    1.0.12 rc5

  • Have you tried reproducing the issue with latest release?
    yes

please read it from bottom to top:

|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|created by google.golang.org/grpc.(*Server).serveStreams.func1|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|/ext-go/1/src/google.golang.org/grpc/server.go:680 +0x9f|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc1993d5f90, 0xc000130380, 0x1595340, 0xc054d94e00, 0xc0fc793900)|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|/ext-go/1/src/google.golang.org/grpc/server.go:1249 +0x1308|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|google.golang.org/grpc.(*Server).handleStream(0xc000130380, 0x1595340, 0xc054d94e00, 0xc0fc793900, 0x0)|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|/ext-go/1/src/google.golang.org/grpc/server.go:1011 +0x4cd|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|google.golang.org/grpc.(*Server).processUnaryRPC(0xc000130380, 0x1595340, 0xc054d94e00, 0xc0fc793900, 0xc00047f1d0, 0x1ed05b8, 0x0, 0x0, 0x0)|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|/ext-go/1/src/github.com/dgraph-io/dgo/protos/api/api.pb.go:1749 +0x23e|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|github.com/dgraph-io/dgo/protos/api._Dgraph_Query_Handler(0x13c1c20, 0x1f9a2d0, 0x158fba0, 0xc1460dc960, 0xc168324e00, 0x0, 0x0, 0x0, 0xc0fc793900, 0x1)|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|/ext-go/1/src/github.com/dgraph-io/dgraph/edgraph/server.go:495 +0x4bf|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|github.com/dgraph-io/dgraph/edgraph.(*Server).Query(0x1f9a2d0, 0x158fba0, 0xc1460dc9c0, 0xc16d46a410, 0xc0f1df2540, 0x0, 0x0)|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|/ext-go/1/src/github.com/dgraph-io/dgraph/query/query.go:2614 +0x5e|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|github.com/dgraph-io/dgraph/query.(*QueryRequest).Process(0xc0149a5a30, 0x158fba0, 0xc1460dc9c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|/ext-go/1/src/github.com/dgraph-io/dgraph/query/query.go:2520 +0x61d|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|github.com/dgraph-io/dgraph/query.(*QueryRequest).ProcessQuery(0xc0149a5a30, 0x158fba0, 0xc1460dc9c0, 0x1, 0x1)|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|/ext-go/1/src/github.com/dgraph-io/dgraph/query/query.go:1557 +0x39|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|github.com/dgraph-io/dgraph/query.(*SubGraph).recursiveFillVars(0xc0d5250840, 0xc1460dd9b0, 0xc0f1df3da0, 0x1)|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|/ext-go/1/src/github.com/dgraph-io/dgraph/query/query.go:1617 +0x249|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|github.com/dgraph-io/dgraph/query.(*SubGraph).fillVars(0xc0d5250840, 0xc1460dd9b0, 0x1, 0xc0000e3b77)|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|/ext-go/1/src/github.com/dgraph-io/dgraph/query/query.go:1646 +0x3e9|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|github.com/dgraph-io/dgraph/query.(*SubGraph).replaceVarInFunc(0xc0d5250840, 0xc1460dd9b0, 0xc0000e3b77)|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|/ext-go/1/src/github.com/dgraph-io/dgraph/types/conversion.go:349 +0x108a|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|github.com/dgraph-io/dgraph/types.Marshal(0x0, 0x0, 0x0, 0xc0149a5320, 0xc0000e3b34, 0xa)|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|goroutine 28820 [running]:|
|Feb 05 22:58:01.826|i-0dd4493d3f656b465|dgraph-alpha|panic: interface conversion: interface {} is nil, not string|
|Feb 05 22:58:01.816|i-0dd4493d3f656b465|dgraph-alpha|Lastcommit 30161 > current 30074. This would cause some commits to be lost.|
|Feb 05 22:58:01.816|i-0dd4493d3f656b465|dgraph-alpha|Lastcommit 30161 > current 30074. This would cause some commits to be lost.|```

@danielmai

This comment has been minimized.

Copy link
Member

danielmai commented Feb 6, 2019

Can you share the steps needed to result in this panic?

  • Minimal sample data set
  • Queries or mutations or other steps that lead to the issue.

@danielmai danielmai added the bug label Feb 6, 2019

@srfrog srfrog self-assigned this Feb 6, 2019

@srfrog

This comment has been minimized.

Copy link
Contributor

srfrog commented Feb 6, 2019

@makitka2007 how can we reproduce it? Thanks.

@makitka2007

This comment has been minimized.

Copy link
Author

makitka2007 commented Feb 7, 2019

unfortunately i can't share steps to reproduce it, it's too complicated, we didn't run 1 mutation and got a panic. we were sending data to dgraph in ~5.000 triples mutation batches based on kafka messages + performed lookups. on 1.0.10 it worked. on 1.0.12-rc5 it crashed after few minutes of work.

may be you have some clue based on stack trace..

@srfrog

This comment has been minimized.

Copy link
Contributor

srfrog commented Feb 7, 2019

No worries, I understand. Yes I will use the stack trace.

@srfrog

This comment has been minimized.

Copy link
Contributor

srfrog commented Feb 8, 2019

@makitka2007 I have PR that I believe is a fix for the issue but I haven't found a query that will trigger the crash. Do you mind checking on your end if that branch fixes it?
#2995

@makitka2007

This comment has been minimized.

Copy link
Author

makitka2007 commented Feb 8, 2019

thank you @srfrog. we have complex deployment process, so we can test only if there will be some tag on docker like rc6/whatever, so will wait for it to test

@srfrog

This comment has been minimized.

Copy link
Contributor

srfrog commented Feb 14, 2019

Merged fix to master. Please reopen if you still having issues.

@srfrog srfrog closed this Feb 14, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment