Skip to content

Commit

Permalink
Rewrite: Drop Bootstrap.jl and use Random API (#7)
Browse files Browse the repository at this point in the history
  • Loading branch information
devmotion committed Apr 26, 2021
1 parent ed1d003 commit fc2c946
Show file tree
Hide file tree
Showing 28 changed files with 1,188 additions and 252 deletions.
1 change: 1 addition & 0 deletions .JuliaFormatter.toml
@@ -0,0 +1 @@
style="blue"
2 changes: 1 addition & 1 deletion .github/workflows/CI.yml
Expand Up @@ -14,7 +14,7 @@ jobs:
fail-fast: false
matrix:
version:
- '1.1'
- '1.0'
- '1'
- 'nightly'
os:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/CompatHelper.yml
Expand Up @@ -18,7 +18,7 @@ jobs:
- name: "Run CompatHelper"
run: |
import CompatHelper
CompatHelper.main()
CompatHelper.main(; subdirs=["", "test", "docs"])
shell: julia --color=yes {0}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Expand Down
27 changes: 27 additions & 0 deletions .github/workflows/Docs.yml
@@ -0,0 +1,27 @@
name: Docs

on:
push:
branches:
- main
tags: '*'
pull_request:

jobs:
docs:
name: Documentation
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: julia-actions/setup-julia@v1
with:
version: '1'
- run: |
julia --project=docs -e '
using Pkg
Pkg.develop(PackageSpec(path=pwd()))
Pkg.instantiate()'
- run: julia --project=docs docs/make.jl
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }}
26 changes: 26 additions & 0 deletions .github/workflows/DocsPreviewCleanup.yml
@@ -0,0 +1,26 @@
name: DocsPreviewCleanup

on:
pull_request:
types: [closed]

jobs:
cleanup:
runs-on: ubuntu-latest
steps:
- name: Checkout gh-pages branch
uses: actions/checkout@v2
with:
ref: gh-pages
- name: Delete preview and history + push changes
run: |
if [ -d "previews/PR$PRNUM" ]; then
git config user.name "Documenter.jl"
git config user.email "documenter@juliadocs.github.io"
git rm -rf "previews/PR$PRNUM"
git commit -m "delete preview"
git branch gh-pages-new $(echo "delete history" | git commit-tree HEAD^{tree})
git push --force origin gh-pages-new:gh-pages
fi
env:
PRNUM: ${{ github.event.number }}
20 changes: 20 additions & 0 deletions .github/workflows/Format.yml
@@ -0,0 +1,20 @@
name: Format

on:
pull_request:

jobs:
format:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: julia-actions/setup-julia@latest
with:
version: 1
- run: |
julia -e 'using Pkg; Pkg.add("JuliaFormatter")'
julia -e 'using JuliaFormatter; format("."; verbose=true)'
- uses: reviewdog/action-suggester@v1
with:
tool_name: JuliaFormatter
fail_on_error: true
2 changes: 2 additions & 0 deletions .gitignore
Expand Up @@ -3,3 +3,5 @@
*.jl.mem
.DS_Store
/Manifest.toml
/test/Manifest.toml
/docs/build/
13 changes: 2 additions & 11 deletions Project.toml
@@ -1,21 +1,12 @@
name = "ConsistencyResampling"
uuid = "4937dc1f-c7a3-5772-9d42-4a8277f2eb51"
authors = ["David Widmann <david.widmann@it.uu.se>"]
version = "0.2.1"
version = "0.3.0"

[deps]
Bootstrap = "e28b5b4c-05e8-5b66-bc03-6f0c0a0a06e0"
Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"
StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"

[compat]
Bootstrap = "2.2"
StatsBase = "0.32, 0.33"
julia = "1.1"

[extras]
SafeTestsets = "1bc83da4-3b8d-516f-aca4-4fe02f6d838f"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["SafeTestsets", "Test"]
julia = "1"
42 changes: 3 additions & 39 deletions README.md
Expand Up @@ -2,49 +2,13 @@

Consistency resampling of calibrated predictions.

[![Stable](https://img.shields.io/badge/docs-stable-blue.svg)](https://devmotion.github.io/ConsistencyResampling.jl/stable)
[![Dev](https://img.shields.io/badge/docs-dev-blue.svg)](https://devmotion.github.io/ConsistencyResampling.jl/dev)
[![Build Status](https://github.com/devmotion/ConsistencyResampling.jl/workflows/CI/badge.svg?branch=main)](https://github.com/devmotion/ConsistencyResampling.jl/actions/workflows/CI.yml?query=branch%3Amain)
[![DOI](https://zenodo.org/badge/186521141.svg)](https://zenodo.org/badge/latestdoi/186521141)
[![Codecov](https://codecov.io/gh/devmotion/ConsistencyResampling.jl/branch/main/graph/badge.svg)](https://codecov.io/gh/devmotion/ConsistencyResampling.jl)
[![Coveralls](https://coveralls.io/repos/github/devmotion/ConsistencyResampling.jl/badge.svg?branch=main)](https://coveralls.io/github/devmotion/ConsistencyResampling.jl?branch=main)

## Overview

This package implements consistency resampling in the Julia language, as described by [Bröcker and Smith (2007)](https://doi.org/10.1175/WAF993.1).
It is based on the [`Bootstrap.jl`](https://github.com/juliangehring/Bootstrap.jl) package for statistical bootstrapping in Julia.

Consistency resampling is a resampling technique that generates calibrated predictions from a data set of predictions
and corresponding labels. First a set of predictions is sampled from the data set with replacement. In a second step
artificial labels are sampled with the predicted probabilities. This resampling procedure ensures that the predictions
are calibrated for the artificial labels.

## Example

The predictions have to be provided as a matrix of size `(m, n)`, in which each of the `n` columns corresponds to
predicted probabilities of the labels `1,…,m`. The corresponding labels have to be provided as a vector of length `n`,
in which every element is from the set `1,…,m`.

```julia
using Distributions

predictions = rand(Dirichlet(10, 1), 500)
labels = rand(1:10, 500)
```

Consistency resampling is performed similar to the other bootstrapping approaches in `Bootstrap.jl`. A random number
generator can be provided as optional argument.

```julia
using ConsistencyResampling
using Distances
using Flux: onehotbatch

b = bootstrap((predictions, labels), ConsistentSampling(100_000)) do (x, y)
totalvariation(x, onehotbatch(y, 1:10)) / 500
end
```

The bootstrapped samples can be explored and used for estimation of confidence intervals, as explained
in the documentation of `Bootstrap.jl`.
[![Code Style: Blue](https://img.shields.io/badge/code%20style-blue-4495d1.svg)](https://github.com/invenia/BlueStyle)

## References

Expand Down

2 comments on commit fc2c946

@devmotion
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JuliaRegistrator
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Registration pull request created: JuliaRegistries/General/35321

After the above pull request is merged, it is recommended that a tag is created on this repository for the registered package version.

This will be done automatically if the Julia TagBot GitHub Action is installed, or can be done manually through the github interface, or via:

git tag -a v0.3.0 -m "<description of version>" fc2c946eb92804644c51acb932f8ca57349a9383
git push origin v0.3.0

Please sign in to comment.