Skip to content
This repository has been archived by the owner on Feb 21, 2024. It is now read-only.

Support keys on import CLI. #1501

Merged
merged 1 commit into from Jul 24, 2018

Conversation

benbjohnson
Copy link
Contributor

@benbjohnson benbjohnson commented Jul 13, 2018

Overview

This pull request adds support for specifying string keys with pilosa import. There was an existing partial implementation but it did not work and required both the index and field to use string keys.

Keys are translated server-side so performance will be limited by the translation file speed. Previous tests show the speed to be quite high though.

Usage

Given an index i, a field f, and the following data.csv file:

FOO,BAR
FOO,BAZ

The import can be run using the normal import command:

$ pilosa import -i i -f f data.csv

Fixes #1488

Pull request checklist

Code review checklist

This is the checklist that the reviewer will follow while reviewing your pull request. You do not need to do anything with this checklist, but be aware of what the reviewer will be looking for.

  • Ensure that any changes to external docs have been included in this pull request.
  • If the changes require that minor/major versions need to be updated, tag the PR appropriately.
  • Ensure the new code is properly commented and follows Idiomatic Go.
  • Check that tests have been written and that they cover the new functionality.
  • Run tests and ensure they pass.
  • Build and run the code, performing any applicable integration testing.

if err != nil {
return errors.Wrap(err, "getting field")
}

// Translate row keys.
if field.keys() {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should validate in this case that the user hasn't passed row IDs, and similarly in the next block that the user hasn't passed any column IDs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I added validation in 7a5b7c3.

Copy link
Member

@jaffee jaffee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good other than my one comment. I made a PR with some tests into your branch which I think would be good to get in before merging.

jaffee
jaffee previously approved these changes Jul 16, 2018
@benbjohnson
Copy link
Contributor Author

@jaffee Sorry, I missed some the XOR logic within Bits.RowIDs() & Bits.ColumnIDs(). Can you review and reapprove?

@jaffee
Copy link
Member

jaffee commented Jul 16, 2018

@benbjohnson I forgot I wrote some tests when I was reviewing this - could you include them? benbjohnson@df91f81

@benbjohnson
Copy link
Contributor Author

@jaffee np. Added in 490a8cb.

@jaffee
Copy link
Member

jaffee commented Jul 23, 2018

ping @benbjohnson - this ok for merge?

@benbjohnson benbjohnson merged commit 214c667 into FeatureBaseDB:develop Jul 24, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants