Skip to content

Issue 2889: Support loading JSON data with live loader #2923

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 48 commits into from

Conversation

codexnull
Copy link
Contributor

@codexnull codexnull commented Jan 22, 2019

This change set add support for loading JSON-formatted data using the live loader. Changes:

  • Accept *.json(.gz) files in addition to *.rdf(.gz).
  • Accept a JSON stream instead of a file.
  • Automatically detect and handle compressed input instead of relying on filename extension.
  • Change many "rdfs" references to "n-quads" since that is more correct and the use of "rdf" with JSON files is potentially confusing.
  • Move some common loading functionality from live package to x so that it may be more easily shared with bulk loader.
  • Add z package to hold code that may be re-used by other dgraph tests.

This change is Reviewable

Javier Alvarado added 28 commits January 11, 2019 16:11
// overwritten, but the attacker could accomplish the same thing by simply omitting the over-
// written data from the load files in the first place.
md := md5.New()
io.WriteString(md, str)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error return value of io.WriteString is not checked (from errcheck)

if opt.keyFields == "" {
return fmt.Errorf("No uid field found: %+v\n", nqs)
}
assignBlankId(nqs)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error return value of assignBlankId is not checked (from errcheck)

}
}

cmd[i].Start()

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Error return value of (*os/exec.Cmd).Start is not checked (from errcheck)

@@ -21,6 +21,7 @@ import (
"bytes"
"compress/gzip"
"context"
"crypto/md5"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

G501: Blacklisted import crypto/md5: weak cryptographic primitive (from gosec)

// here. If an "attacker" crafted a JSON file such that a collision occurs some data may be
// overwritten, but the attacker could accomplish the same thing by simply omitting the over-
// written data from the load files in the first place.
md := md5.New()

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

G401: Use of weak cryptographic primitive (from gosec)

}

var opt options
var tlsConf x.TLSHelperConfig

var Live x.SubCommand

var keyFields []string

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File is not goimports-ed (from goimports)

@codexnull codexnull closed this Feb 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants