@b5 b5 released this Oct 10, 2018

Assets 3

👭 Peer Sharing Fixes & better export

This patch release is focused on making inroads on some long-standing issues with peer discovery. We've added tests & a few fixes that should help peers get connected and stay connected. This is an area of active work that we'll be adding more improvements to in the future.

We've also standardized our export format with a newly-approved export rfc, which fixes a few issues with the way Qri exports .zip archives. exports should now be complete archives that we'll start using for import in a forthcoming release.

@b5 b5 released this Oct 5, 2018 · 22 commits to master since this release

Assets 3

Version 0.5.5 is a patch release with some small features and a few bugfixes. It's mainly here because @b5 wants
to play with .zip files & regexes in transform scripts.

📦 Skylark re and zip packages

We've added two new small, bare-bones packages to skylark to handle common-yet-vital use cases:
re brings basic support regular expressions, and zip brings read-only capacity to open zip archives.
Both of these are rather utility-oriented, but very importnat when opening & cleaning data.

🔀 Upcoming switch from "skylark" to "starlark"

Speaking of skylark, google has landed on a rename for thier project, and it'll hence-fourth be named "starlark".
As such we'll be making the switch to this terminology in an upcoming release. Our package names will be changing
from .sky to some new file extension, which will be a breaking change for all tranforms that import .sky packages.
We'll keep you posted.

@b5 b5 released this Oct 1, 2018 · 37 commits to master since this release

Assets 3

🔧 minor patch release

0.5.4 is a very minor release that cleans up a few issues with our API to make the frontend editor work 😄

@b5 b5 released this Sep 24, 2018 · 43 commits to master since this release

Assets 2

🏃‍♀️ Dry Run

0.5.3 is a minor version bump that introduces a new --dry-run flag on qri new and qri save that'll run new/save without committing changes. We've added the same thing on the api using a query param: dry_run=true. Dry runs are fun! use 'em to experiment with different input without having to constantly delete stuff.

@b5 b5 released this Sep 14, 2018 · 58 commits to master since this release

Assets 3

Most of 0.5.2 is under-the-hood changes that make qri work better. We've put a lot of time into our test suite & refactoring our existing code, which will set the stage for dev work to move a little faster in the coming weeks 😄.

🚚 IPFS repo migration

We've updated our IPFS dependencies from go-ipfs 0.4.15 to 0.4.17. Between those verions the structure of IPFS repositories changed, and as such need to be migrated. If this is your first time installing Qri, you won't have an issue, but if you're coming from Qri 0.5.1 or earlier you'll see a message after upgrading that'll explain you'll need to install go-ipfs to update your repo. We know this is, well, not fun, but it's the safest way to ensure the integrity of your underlying IPFS repo.

@b5 b5 released this Jul 19, 2018 · 171 commits to master since this release

Assets 3

Ok ok so now we have a formal 0.5.1 release. Maybe this should be 0.6.0 given the magnitude of visualizations, but meh, we're calling it a patch.

📊 Delight in Data with HTML-template visualizations

For a little while we've been experimenting with the qri render as a way to template data into html. The more we've played with it, the more we've come to rely on it. So much so, that we think templates should become a native component of datasets. For this we've added a new section to the dataset definition called viz, which is where you specify a custom template. a dataset.yaml file that specifies viz will look something like this (These details are always avilable with the handy qri export --blank):

# use viz to provide custom a HTML template of your dataset
# the currently accepted syntax is 'html'
# scriptpath is the path to your template, relative to this file:
# viz:
#   syntax: html
#   scriptpath: template.html

A template can only be a single file. So your template can't reference a local "style.css" file for now. But! You're more than welcome to write HTML that grabs external resources, like getting jQuery or D3, or your favourite charting library off a CDN.

When you supply a template, the Dataset itself is handed back to you, and we use golang's html/template package to render. Here's a simplified example:

    <h1>{{ .Meta.Title }}</h1>
    <p>{{ .Meta.Description }}</p>
    {{ range .Body }}
      <li>{{ . }}</li>
    {{ end }}

This would print the Title & Description from your dataset's metadata. If your dataset was a list of strings, each entry would be printed out as a list item.

Moar Better Frontend

We're so into viz, we've completely overhauled the frontend to put custom templates front-and-center for datasets. Be sure to take the new frontend for a spin by visiting http://localhost:2505 while qri connect is running.

We've chosen to invest time in viz because we think it brings an important moment of delight to datasets. Without it, all of this data stuff just feels like work. There's something truly magical about seeing your data rendered in a custom page that makes all the munging worthwhile.


@b5 b5 released this Jul 14, 2018 · 187 commits to master since this release

Assets 3

0.5.1 is a release candidate to get a few important details out the door to demo at the wonderful our networks conference in Toronto. we antipate further changes before cutting a proper 0.5.1 release, which we'll properly document. More soon!

@b5 b5 released this Jun 18, 2018 · 250 commits to master since this release

Assets 3

Who needs patch releases!? In version 0.5.0 we're introducing an initial search implementation, a few new commands, rounding out a bunch of features that warrent a minor version bump with some breaking changes. Because of these breaking changes, datasets created with v0.4.0 or earler will need to be re-created to work properly with this version.

🔍 Registry Search Alpha

We're still hard at work on getting registries right (more on that in the coming weeks), but for now we've shipped an initial command "qri search", that'll let you search registries for datasets. The way we see search working in the future is leveraging registries to build indexes of published datasets so you can easily search for datasets that have been published. We have a lot of work to do around making sure those datasets are available for download, but feel free to play with the command to get a feel for where we're headed with search.

use & get commands

working with the command line, it can get really irritating to constantly re-type the name of a dataset. To help with this, we've added a new command: qri use, which takes it's inspiration from database "use" commands that set the current database. qri use takes any number of dataset references as arguments, and once a user has set a selection with qri use, they become the default dataset names when no selection is made.

Around qri HQ we've all come to love the ease of working with the qri config command. qri config get shows either the whole configuration, or you can provide a dot.separated.path to scope down the section of config to show. qri get takes this idea an applies it to datasets. qri get meta.title me/dataset_name will get the title from metadata, and like the config command, it's output can be switched from YAML (default) to JSON. qri get also accepts multiple datasets, which will be combined into

new skylark html module

We're still working on a proper html module for skylark transforms with an API that pythonists will be familiar with, but in the meantime we've added in a basic jquery-like selector syntax for working with HTML documents.

"Data" is now "Body"

This is a breaking change we've been hoping to get in sooner-rather-than-later that renames the Data field of a dataset to Body. From here on in we'll refer to the body of a dataset as it's principle content. We think this langauge helps show how datasets are like webpages, and cutw down on use of an ambiguous term like "data".

Thinking about qri as an importable library

Finally, in what is more of a symbolic change than anything else, we've renamed the core package to lib to get us to start thinking about qri as an importable library. Next week we'll publish a new project aimed at writing tutorials & docs with an associated test suite built around datasets that uses qri as a library. We hope to use this project to mature the lib package for this use case.

@b5 b5 released this Jun 6, 2018 · 291 commits to master since this release

Assets 3

We're going to start writing proper release notes now, so, uh, here are those notes:

This release brings a big new feature in the form of our first transformation implementation, and a bunch of refinements that make the experience of working with qri a little easier.

Introducing Skylark Transformations

For months qri has had a planned feature set for embedding the concept of "transformations" directly into datasets. Basically transforms are scripts that auto-generate datasets. We've defined a "transformation" to be a repeatable process that takes zero or more datasets and data sources, and outputs exactly one dataset. By embedding transformations directly into datasets, users can repeat them with a single command, keeping the code that updates a dataset, and it's resulting data in the same place. This opens up a whole new set of uses for qri datasets, making them auditable, repeatable, configurable, and generally functional. Using transformations, qri can check to see if your dataset is out of date, and update it for you.

While we've had the plan for transformations for some time now, it's taken us a long time to figure out how to write a first implementaion. Because transformations are executable code, security & behavioural expectations are a big concern. We also want to set ourselves up for success by choosing an implementation that will feel familiar to those who do a lot of code-based data munging, while also leaving the door open to things we'd like to do in the future like parallelized execution.

So after a lot of reasearch and a false-start or five, we've decided on a scripting language called skylark as our base implementation, which has grown out of the bazel project at google. This choice might seem strange at first (bazel is a build tool and has nothing to do with data), but skylark has a number of advantages:

  • python-like syntax - many people working in data science these days write python, we like that.
  • deterministic subset of python - unlike python, skylark removes properties that reduce introspection into code behaviour. things like while loops and recursive functions are ommitted, making it possible for qri to infer how a given transformation will behave.
  • parallel execution - thanks to this deterministic requirement (and lack of global interpreter lock) skylark functions can be executed in parallel. Combined with peer-2-peer networking, we're hoping to advance tranformations toward peer-driven distribed computing. More on that in the coming months.

A tutorial on how to write skylark transformations is forthcoming, we'll post examples to our documentation site when it's ready: https://qri.io/docs

dataset.yaml, and more ❤️ for the CLI

For a while now we've been thinking about datasets as being a lot like web pages. Web pages have head,meta and body elements. Datasets have meta, structure, commit, and data. To us this metaphor helps reason about the elements of a dataset, why they exist, and their function. And just like how webpages are defined in .html files, we've updated the CLI to work with .yaml files that define dataests. qri export --blank will now write a blank dataset file with comments that link to documentation on each section of a dataset. You can edit that file, save it, and run qri add --dataset=dataset.yaml me/my_dataset to add the dataset to qri. Ditto for qri update.

We'd like to encourage users to think in terms of these dataset.yaml files, building up a mental model of each element of a dataset in much the same way we think about HTML page elements. We chose yaml over JSON specifically because we can include comments in these files, making it easier to pass them around with tools outside of qri, and we're hoping this'll make it easier to think about datasets moving forward. In futures release we plan to rename the "data" element to "body" to bring this metaphor even closer.

Along with dataset.yaml, we've also done a bunch of refactoring & bug fixes to make the CLI generally work better, and look forward to improving on this trend in near-term patch releases. One of the biggest things we'd like to improve upon is providing more meaningful error messages.

@b5 b5 released this May 15, 2018 · 355 commits to master since this release

Assets 3
chore(release): release v0.3.2

Merge pull request #402 from qri-io/v032