Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
bin
 
 
exe
 
 
lib
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

README.md

BqGuess

Gem Version Code Climate Test Coverage Issue Count

BqGuess guesses BigQuery schema from existing logs.

Installation

$ gem install bq_guess

Usage

$ bq_guess existing_log.jsonl

For example:

$ cat existing_log.jsonl
{"required":123,"optional":true,"nested":{"required":1234,"optional":"yes"},"array":[0,1,2,3,4]}
{"required":456,"optional":false,"nested":{"required":1234,"optional":"yes","nested":{"prop":1}},"array":[5,6,7,8,9]}
{"required":789,"nested":{"required":1234,"optional":"yes","additional":"added"},"array":[]}

$ bq_guess existing_log.jsonl
[
  {
    "name": "required",
    "type": "INTEGER",
    "mode": "REQUIRED"
  },
  {
    "name": "optional",
    "type": "BOOLEAN",
    "mode": "NULLABLE"
  },
  {
    "name": "nested",
    "type": "RECORD",
    "mode": "REQUIRED",
    "fields": [
      {
        "name": "required",
        "type": "INTEGER",
        "mode": "REQUIRED"
      },
      {
        "name": "optional",
        "type": "STRING",
        "mode": "REQUIRED"
      },
      {
        "name": "nested",
        "type": "RECORD",
        "mode": "NULLABLE",
        "fields": [
          {
            "name": "prop",
            "type": "INTEGER",
            "mode": "REQUIRED"
          }
        ]
      },
      {
        "name": "additional",
        "type": "STRING",
        "mode": "NULLABLE"
      }
    ]
  },
  {
    "name": "array",
    "type": "INTEGER",
    "mode": "REPEATED"
  }
]

Supported Formats

  • JSON Lines
  • LTSV

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/nownabe/bq_guess.

License

The gem is available as open source under the terms of the MIT License.

About

Guess BigQuery Schema

Resources

License

Packages

No packages published
You can’t perform that action at this time.