Releasing 0.2.0?

[sunchao]

We made several important changes on the code base recently:

1. Get rid of the Thrift dependency which makes this crate much easier to use by other projects (Integration with parquet-format #82 )
2. Updated documentation and with 1) we will publish the doc to docs.rs, which will be very useful (Update documentation #81)
3. Improved decoding performance for various encodings including dict, rle, delta_binary_packed, etc. (Read batch of values for RLE, DELTA_BINARY_PACKED and PLAIN decoding #80)

I think it may be a good time to publish version 0.2.0.

[andygrove]

I would especially appreciate this as I can no longer publish my documentation to docs.rs because of the the thrift issue.

[sadikovi]

Agree 100%

[sadikovi]

When are we going to publish 0.2.0? Should we wait for #86 and #91, and #94?
I will try fixing statistics tomorrow or after we have settled on the design, and will update PR on decimals.

@andygrove Let me know if you need any help with field accessors PR. Thanks!

[sunchao]

Sorry for the delay @sadikovi . I'm hoping to at least get #86 in since it is a API change.

[andygrove]

@sunchao #86 is good to go (@sadikovi approved it)

[sadikovi]

I think we should try getting the following in the 0.2.0 release:

• Remove features flags that are stablized #97
• Add LZ4 compression #98
• maybe Add statistics for column chunk metadata and data page #94, but I think it is better to work on it a bit more

[sunchao]

I'm also working on a PR to support accessors for List & Map, which is also necessary for 0.2 I think. Otherwise, one pretty much can't do anything after getting List & Map.

[sadikovi]

Nice, thanks!

[sunchao]

All the required PRs are merged, except #94. @sadikovi : do you think we should postpone that to a later release? If so, I'll release 0.2.0 today. 😃

[sadikovi]

Well done! I think we should release 0.2.0. I am still working on #94, and it does not block the release and it will take me some time to finish.

Currently the crate version is mentioned in lib.rs and readme, so we need to update those as well.

[sunchao]

I just published 0.2.0! However, docs.rs did not use nightly rustc and so failed to compile the project. I'm figuring out a way to fix this now. We may need to publish a minor release for the doc issue.

[andygrove]

Another option is running cargo docs manually and self hosting the docs.

…

On Wed, Apr 25, 2018, 10:12 PM Chao Sun ***@***.***> wrote: I just published 0.2.0! However, docs.rs did not use nightly rustc and so failed to compile the project <https://docs.rs/crate/parquet/0.2.0/builds/97194>. I'm figuring out a way to fix this now. We may need to publish a minor release for the doc issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#88 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA5AxN7cbXk1MVgOSjBQe-NVd7ySv0Yuks5tsUk5gaJpZM4TU4YR> .

[sunchao]

@andygrove Thanks. Will take that as an option also.

[sadikovi]

@sunchao Would you mind creating a new release on Github (https://github.com/sunchao/parquet-rs/releases)? Thanks!

Let me know if you need any help with publishing docs on Github pages. I am more than happy to help out.

[sunchao]

@sadikovi . Sure. I just published the release. I'm still looking at why docs.rs doesn't use nightly for build. Yeah, it would be great if you can also look at publishing docs to Github pages. Thanks!

[sadikovi]

I thought it would be acceptable to do the following:

• Having /docs folder the main repo or having a separate repository for docs
• Having a script that we run when releasing a new version, that script will do:
  • Check that there are no outstanding changes in master branch
  • Fetch the list of tags, validate the new version
  • Build the crate, run tests optionally
  • Build docs
  • Change all places for the new version, move into /docs folder
  • Commit all changes, tag the commit
  • Cargo publish
  • Push changes, including the tag

The idea is running the script for every release, documentation will be published for every release manually, instead of publishing on every commit in travis. Each release will be stored in repository like this:

/docs/0.1.0/...
/docs/0.2.0/...
/docs/[tag]/...

See an example here: https://github.com/sadikovi/parquet-rs-docs/tree/master/docs

Available:
https://sadikovi.github.io/parquet-rs-docs/v0.1.0/parquet/
https://sadikovi.github.io/parquet-rs-docs/v0.2.0/parquet/

This has the following benefits:

• Will create a certain process for publishing the crate
• Will enforce commit message, tags format, etc.
• Easy to publish

We can use cargo make task runner for it (https://github.com/sagiegurari/cargo-make).

Or as an alternative Travis supports pushing to Github pages: https://docs.travis-ci.com/user/deployment/pages/. This would mean that we just need to add cargo doc as a build step and configure deploy step in Travis. But this would build docs for every commit to master.

Let me know what you think.

[sadikovi]

Or as an alternative Travis supports pushing to Github pages: https://docs.travis-ci.com/user/deployment/pages/. This would mean that we just need to add cargo doc as a build step and configure deploy step in Travis. But this would build docs for every commit to master.

[sunchao]

@sadikovi Thanks for the suggestions. Personally I'm in favor of the github pages approach, for deploying docs for master branch. I think we can create a branch gh-pages and publish docs there. It should be OK to build docs for every commit (not including pull requests) as it is relatively fast. A good example will be https://github.com/hyperium/hyper.

On the other hand, I'm in favor of publishing docs for different versions on docs.rs. I've filed an issue there, and it seems we only need to revert #97 to make the build successful. But I think we should wait until they update the binary to use nightly.

[sadikovi]

Let's settle on gh-pages for now. I was thinking if publishing manually is better. By manual publishing, I mean having a script that updates new version, publishes crate and uploads docs on gh-pages branch.

But we could do it either way.

[sadikovi]

I will play with it in my test repository and report back. Cheers!

[sadikovi]

I played with publishing docs on gh-pages. I explain what I tried and then the actual steps to add it.

Approaches

First, I tried publishing docs via deploy feature on Travis (it supports GitHub pages). It does work nicely, except the fact that git push is always force, which overrides the root directory, so we could only keep the latest documentation folder. I have all the settings in my test repo (see https://github.com/sadikovi/parquet-rs-docs/blob/d3a41f99f62a77bddb42a4dc61ee891613410ccc/.travis.yml)

Then, I decided to go with @sunchao suggested approach similar to hyper repo. Though the solution was inspired by it, I did not actually use any of the files from hyper, and wrote script myself.

This solution does work on master and tags, so documentation is automatically published for master
on every commit, and for every created tag on GitHub (or pushed from local, which is what I normally do). This allows us to keep versions of documentation, see below.

All examples of setup are here:
https://github.com/sadikovi/parquet-rs-docs (master branch)
https://github.com/sadikovi/parquet-rs-docs/tree/gh-pages (gh-pages branch)

You can also click on badges, links redirect to the correct versions.

Steps to make it work

1. Create empty gh-pages branch, see the code below:

git checkout --orphan gh-pages
git rm -rf .
rm -f .gitignore
rm -f Cargo.lock
git commit --allow-empty -m "root commit"
git push origin gh-pages

2. Check out the gh-pages branch and add .gitignore and README.md.
   Content should be like this:

   • https://github.com/sadikovi/parquet-rs-docs/blob/gh-pages/.gitignore
   • https://github.com/sadikovi/parquet-rs-docs/blob/gh-pages/README.md

3. Commit and push changes to gh-pages. The rest of the steps is done on master branch.

4. Check out master branch and make the following changes:

   • add .travis folder and deploy-docs.sh file (https://github.com/sadikovi/parquet-rs-docs/blob/master/.travis/deploy-docs.sh).
   • if you want, you can configure user/email for the commit, etc.
   • run chmod u+x ./.travis/deploy-docs.sh to make it exec.
   • add the following to .travis.yml (pasted as diff, so you can see the changes):

diff --git a/.travis.yml b/.travis.yml
index 94f1d1e..dd256f9 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -19,12 +19,16 @@ rust:
 script:
   - cargo build
   - cargo test
+  - cargo doc --no-deps
 
 after_success:
   - if [ "$TRAVIS_RUST_VERSION" == "nightly" ]; then
         cargo bench;
     fi
   - cargo coveralls --verbose --exclude-pattern '/parquet-rs/src/bin'
+  - '[ "$TRAVIS_PULL_REQUEST" = false ] &&
+    { [ "$TRAVIS_TAG" != "" ] || [ "$TRAVIS_BRANCH" = "master" ]; } &&
+    ./.travis/deploy-docs.sh'
 
 env:
   global:

4. Commit changes and push changes, we are almost done.
5. Create personal access token on GitHub (somewhere in settings) with the scope "public_repo" only.
6. Go to Travis CI repo settings console and add GITHUB_TOKEN token variable with the token as value. Make sure that "display value in logs" is off (which is default anyway).
7. Retrigger build (either manually or with another commit), now if everything is okay, it will push docs onto gh-pages.

We can also add badges to README, one for master docs and another for the latest published version. Unfortunately, we would have to manually update version in README for badge, but it is a minor thing (see the actual links below):

[![API docs](https://img.shields.io/badge/docs-v0.4.1-blue.svg)](https://sadikovi.github.io/parquet-rs-docs/v0.4.1/parquet/)
[![Master API docs](https://img.shields.io/badge/docs-master-green.svg)](https://sadikovi.github.io/parquet-rs-docs/master/parquet/)

See an example in test repo: https://github.com/sadikovi/parquet-rs-docs/blob/master/README.md

We could update everything in README when creating a "publish" commit.

That is pretty much it. If any questions, refer to the test repo: https://github.com/sadikovi/parquet-rs-docs.

Note that every time we pull, gh-pages is updated, so it might be a little bit annoying in terms
of pulling changes from remote. Apart from that everything should be okay.

[sadikovi]

@sunchao Let me know what you think of the approach, and let me know if you want to do it, or I should make the changes for docs (I can open a PR). Thanks!

[sunchao]

Thanks @sadikovi . This is excellent description! I'll try to make changes following this.

[sadikovi]

I think we should close this issue as 0.2.0 has been released including docs.
Could we release 0.3.0? I think there have been a considerable number of changes that could qualify for 0.3.0. Thanks.

[sunchao]

@sadikovi Sure. I'll release it today.