Skip to content

Commit

Permalink
update v2.0-pre (#953)
Browse files Browse the repository at this point in the history
* Update doc URL. (#821)

* Support indexing 2-axes RaggedTensor, Support slicing for RaggedTensor (#825)

* Support index 2-axes RaggedTensor, Support slicing for RaggedTensor

* Fix compiling errors

* Fix unit test

* Change RaggedTensor.data to RaggedTensor.values

* Fix style

* Add docs

* Run nightly-cpu when pushing code to nightly-cpu branch

* Prune with max_arcs in IntersectDense (#820)

* Add checking for array constructor

* Prune with max arcs

* Minor fix

* Fix typo

* Fix review comments

* Fix typo

* Release v1.8

* Create a ragged tensor from a regular tensor. (#827)

* Create a ragged tensor from a regular tensor.

* Add tests for creating ragged tensors from regular tensors.

* Add more tests.

* Print ragged tensors in a way like what PyTorch is doing.

* Fix test cases.

* Trigger GitHub actions manually. (#829)

* Run GitHub actions on merging. (#830)

* Support printing ragged tensors in a more compact way. (#831)

* Support printing ragged tensors in a more compact way.

* Disable support for torch 1.3.1

* Fix test failures.

* Add levenshtein alignment (#828)

* Add levenshtein graph

* Contruct k2.RaggedTensor in python part

* Fix review comments, return aux_labels in ctc_graph

* Fix tests

* Fix bug of accessing symbols

* Fix bug of accessing symbols

* Change argument name, add levenshtein_distance interface

* Fix test error, add tests for levenshtein_distance

* Fix review comments and add unit test for c++ side

* update the interface of levenshtein alignment

* Fix review comments

* Release v1.9

* Support a[b[i]] where both a and b are ragged tensors. (#833)

* Display import error solution message on MacOS (#837)

* Fix installation doc. (#841)

* Fix installation doc.

Remove Windows support. Will fix it later.

* Fix style issues.

* fix typos in the install instructions (#844)

* make cmake adhere to the modernized way of finding packages outside default dirs (#845)

* import torch first in the smoke tests to preven SEGFAULT (#846)

* Add doc about how to install a CPU version of k2. (#850)

* Add doc about how to install a CPU version of k2.

* Remove property setter of Fsa.labels

* Update Ubuntu version in GitHub CI since 16.04 reaches end-of-life.

* Support PyTorch 1.10. (#851)

* Fix test cases for k2.union() (#853)

* Fix out-of-boundary access (read). (#859)

* Update all the example codes in the docs (#861)

* Update all the example codes in the docs

I have run all the modified codes with  the newest version k2.

* do some changes

* Fix compilation errors with CUB 1.15. (#865)

* Update README. (#873)

* Update README.

* Fix typos.

* Fix ctc graph (make aux_labels of final arcs -1) (#877)

* Fix LICENSE location to k2 folder (#880)

* Release v1.11. (#881)

It contains bugfixes.

* Update documentation for hash.h (#887)

* Update documentation for hash.h

* Typo fix

* Wrap MonotonicLowerBound (#883)

* Wrap MonotonicLowerBound

* Add unit tests

* Support int64; update documents

* Remove extra commas after 'TOPSORTED' properity and fix RaggedTensor constructer parameter 'byte_offset' out-of-range bug. (#892)

Co-authored-by: gzchenduisheng <gzchenduisheng@corp.netease.com>

* Fix small typos (#896)

* Fix k2.ragged.create_ragged_shape2 (#901)

Before the fix, we have to specify both `row_splits` and `row_ids`
while calling `k2.create_ragged_shape2` even if one of them is `None`.

After this fix, we only need to specify one of them.

* Add rnnt loss (#891)

* Add cpp code of mutual information

* mutual information working

* Add rnnt loss

* Add pruned rnnt loss

* Minor Fixes

* Minor fixes & fix code style

* Fix cpp style

* Fix code style

* Fix s_begin values in padding positions

* Fix bugs related to boundary; Fix s_begin padding value; Add more tests

* Minor fixes

* Fix comments

* Add boundary to pruned loss tests

* Use more efficient way to fix boundaries (#906)

* Release v1.12 (#907)

* Change the sign of the rnnt_loss and add reduction argument (#911)

* Add right boundary constrains for s_begin

* Minor fixes to the interface of rnnt_loss to make it return positive value

* Fix comments

* Release a new version

* Minor fixes

* Minor fixes to the docs

* Fix building doc. (#908)

* Fix building doc.

* Minor fixes.

* Minor fixes.

* Fix building doc (#912)

* Fix building doc

* Fix flake8

* Support torch 1.10.x (#914)

* Support torch 1.10.x

* Fix installing PyTorch.

* Update INSTALL.rst (#915)

* Update INSTALL.rst

Setting a few additional env variables to enable compilation from source *with CUDA GPU computation support enabled*

* Fix torch/cuda/python versions in the doc. (#918)

* Fix torch/cuda/python versions in the doc.

* Minor fixes.

* Fix building for CUDA 11.6 (#917)

* Fix building for CUDA 11.6

* Minor fixes.

* Implement Unstack (#920)

* Implement unstack

* Remove code does not relate to this PR

* Remove for loop on output dim; add Unstack ragged

* Add more docs

* Fix comments

* Fix docs & unit tests

* SubsetRagged & PruneRagged (#919)

* Extend interface of SubsampleRagged.

* Add interface for pruning ragged tensor.

* Draft of new RNN-T decoding method

* Implements SubsampleRaggedShape

* Implements PruneRagged

* Rename subsample-> subset

* Minor fixes

* Fix comments

Co-authored-by: Daniel Povey <dpovey@gmail.com>

* Add Hash64 (#895)

* Add hash64

* Fix tests

* Resize hash64

* Fix comments

* fix typo

* Modified rnnt (#902)

* Add modified mutual_information_recursion

* Add modified rnnt loss

* Using more efficient way to fix boundaries

* Fix modified pruned rnnt loss

* Fix the s_begin constrains of pruned loss for modified version transducer

* Fix Stack (#925)

* return the correct layer

* unskip the test

* Fix 'TypeError' of rnnt_loss_pruned function. (#924)

* Fix 'TypeError' of rnnt_loss_simple function.

Fix 'TypeError' exception when calling rnnt_loss_simple(..., return_grad=False)  at validation steps.

* Fix 'MutualInformationRecursionFunction.forward()' return type check error for pytorch < 1.10.x

* Modify return type.

* Add documents about class MutualInformationRecursionFunction.

* Formated code style.

* Fix rnnt_loss_smoothed return type.

Co-authored-by: gzchenduisheng <gzchenduisheng@corp.netease.com>

* Support torch 1.11.0 and CUDA 11.5 (#931)

* Support torch 1.11.0 and CUDA 11.5

* Implement Rnnt decoding (#926)

* first working draft of rnnt decoding

* FormatOutput works...

* Different num frames for FormatOutput works

* Update docs

* Fix comments, break advance into several stages, add more docs

* Add python wrapper

* Add more docs

* Minor fixes

* Fix comments

* fix building docs (#933)

* Release v1.14

* Remove unused DiscountedCumSum. (#936)

* Fix compiler warnings. (#937)

* Fix compiler warnings.

* Minor fixes for RNN-T decoding. (#938)

* Minor fixes for RNN-T decoding.

* Removes arcs with label 0 from the TrivialGraph. (#939)

* Implement linear_fsa_with_self_loops. (#940)

* Implement linear_fsa_with_self_loops.

* Fix the pruning with max-states (#941)

* Rnnt allow different encoder/decoder dims (#945)

* Allow different encoder and decoder dim in rnnt_pruning

* Bug fixes

* Supporting building k2 on Windows (#946)

* Fix nightly windows CPU build (#948)

* Fix nightly building k2 for windows.

* Run nightly build only if there are new commits.

* Check the versions of PyTorch and CUDA at the import time. (#949)

* Check the versions of PyTorch and CUDA at the import time.

* More straightforward message when CUDA support is missing (#950)

* Implement ArrayOfRagged (#927)

* Implement ArrayOfRagged

* Fix issues and pass tests

* fix style

* change few statements of functions and move the definiation of template Array1OfRagged to header file

* add offsets test code

* Fix precision (#951)

* Fix precision

* Using different pow version for windows and *nix

* Use int64_t pow

* Minor fixes

Co-authored-by: Fangjun Kuang <csukuangfj@gmail.com>
Co-authored-by: Piotr Żelasko <petezor@gmail.com>
Co-authored-by: Jan "yenda" Trmal <jtrmal@gmail.com>
Co-authored-by: Mingshuang Luo <37799481+luomingshuang@users.noreply.github.com>
Co-authored-by: Ludwig Kürzinger <lumaku@users.noreply.github.com>
Co-authored-by: Daniel Povey <dpovey@gmail.com>
Co-authored-by: drawfish <duisheng.chen@gmail.com>
Co-authored-by: gzchenduisheng <gzchenduisheng@corp.netease.com>
Co-authored-by: alexei-v-ivanov <alexei_v_ivanov@ieee.org>
Co-authored-by: Wang, Guanbo <wgb14@outlook.com>
Co-authored-by: Nickolay V. Shmyrev <nshmyrev@gmail.com>
Co-authored-by: LvHang <hanglyu1991@gmail.com>
  • Loading branch information
13 people committed Apr 14, 2022
1 parent 1d30b9a commit 7f2ca04
Show file tree
Hide file tree
Showing 65 changed files with 1,706 additions and 1,207 deletions.
1 change: 1 addition & 0 deletions .flake8
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ exclude =
get_version.py
build,
k2/python/host,
k2/python/k2/__init__.py,
k2/python/k2/ctc_loss.py,
docs

Expand Down
117 changes: 117 additions & 0 deletions .github/workflows/build-conda-cpu-macos.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Copyright 2021 Xiaomi Corp. (author: Fangjun Kuang)

# See ../../LICENSE for clarification regarding multiple authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# refer to https://github.com/actions/starter-workflows/pull/47/files


# Note, we have to set
#
# export DYLD_LIBRARY_PATH=$CONDA_PREFIX/lib/python3.8/site-packages:$DYLD_LIBRARY_PATH
#
# before running `python3 -m k2.version`
#
# See https://github.com/openPMD/openPMD-api/issues/593#issuecomment-552690470


name: build_conda_cpu_macos

on:
push:
tags:
- '*'

env:
K2_BUILD_TYPE: Release

jobs:
generate_build_matrix:
# see https://github.com/pytorch/pytorch/pull/50633
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Generating build matrix
id: set-matrix
run: |
# outputting for debugging purposes
python scripts/github_actions/generate_build_matrix.py
MATRIX=$(python scripts/github_actions/generate_build_matrix.py)
echo "::set-output name=matrix::${MATRIX}"
build_conda_cpu_macos:
needs: generate_build_matrix
runs-on: macos-10.15
strategy:
fail-fast: false
matrix:
${{ fromJson(needs.generate_build_matrix.outputs.matrix) }}

steps:
# refer to https://github.com/actions/checkout
- uses: actions/checkout@v2
with:
fetch-depth: 0

- uses: conda-incubator/setup-miniconda@v2
with:
auto-update-conda: true
python-version: ${{ matrix.python-version }}
activate-environment: k2

- name: Display Python version
shell: bash -l {0}
run: |
python3 -c "import sys; print(sys.version)"
which python3
- name: Install conda dependencies
shell: bash -l {0}
run: |
conda install -y -q anaconda-client
conda install -y -q conda-build
conda install -y -q -c pytorch pytorch=${{ matrix.torch }} cpuonly
- name: Display conda info
shell: bash -l {0}
run: |
which conda
conda env list
conda info
- name: Build k2
shell: bash -l {0}
env:
K2_PYTHON_VERSION: ${{ matrix.python-version}}
K2_TORCH_VERSION: ${{ matrix.torch }}
K2_CONDA_TOKEN: ${{ secrets.K2_CONDA_TOKEN}}
K2_IS_GITHUB_ACTIONS: 1
K2_IS_FOR_CONDA: 1
run: |
export K2_BUILD_TYPE=$K2_BUILD_TYPE
./scripts/build_conda_cpu.sh
- name: Display generated files
run: |
ls -lh /usr/local/miniconda/envs/k2/conda-bld/osx-64
- name: Upload generated files
uses: actions/upload-artifact@v2
with:
name: torch-${{ matrix.torch }}-python-${{ matrix.python-version }}-${{ matrix.os }}
path: /usr/local/miniconda/envs/k2/conda-bld/osx-64/*.tar.bz2
Original file line number Diff line number Diff line change
Expand Up @@ -26,68 +26,41 @@
# See https://github.com/openPMD/openPMD-api/issues/593#issuecomment-552690470


name: build_conda_cpu
name: build_conda_cpu_ubuntu

on:
push:
branches:
- conda-cpu
tags:
- '*'

env:
K2_BUILD_TYPE: Release

jobs:
build_conda_cpu:
runs-on: ${{ matrix.os }}
generate_build_matrix:
# see https://github.com/pytorch/pytorch/pull/50633
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Generating build matrix
id: set-matrix
run: |
# outputting for debugging purposes
python scripts/github_actions/generate_build_matrix.py
MATRIX=$(python scripts/github_actions/generate_build_matrix.py)
echo "::set-output name=matrix::${MATRIX}"
build_conda_cpu_ubuntu:
needs: generate_build_matrix
runs-on: ubuntu-18.04
strategy:
fail-fast: false
matrix:
os: [ubuntu-18.04, macos-10.15]
python-version: ["3.6", "3.7", "3.8", "3.9", "3.10"]
# from https://download.pytorch.org/whl/torch_stable.html
#
# PyTorch 1.11.x supports 3.7, 3.8, 3.9, 3.10
# PyTorch 1.10, 1.9.x, 1.8.x, and 1.7.1 support 3.6, 3.7, 3.8, 3.9
# PyTorch 1.7.0, 1.6.0, and 1.5.x support 3.6, 3.7, 3.8
#
# Other PyTorch versions are not tested
#
torch: ["1.5.0", "1.5.1", "1.6.0", "1.7.0", "1.7.1", "1.8.0", "1.8.1", "1.9.0", "1.9.1", "1.10.0", "1.10.1", "1.10.2", "1.11.0"]
exclude:
- python-version: "3.9" # exclude Python 3.9 for [1.5.0, 1.5.1, 1.6.0, 1.7.0]
torch: "1.5.0"
- python-version: "3.9"
torch: "1.5.1"
- python-version: "3.9"
torch: "1.6.0"
- python-version: "3.9"
torch: "1.7.0"
- python-version: "3.10" # exclude Python 3.10 for [1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2]
torch: "1.5.0"
- python-version: "3.10"
torch: "1.5.1"
- python-version: "3.10"
torch: "1.6.0"
- python-version: "3.10"
torch: "1.7.0"
- python-version: "3.10"
torch: "1.7.1"
- python-version: "3.10"
torch: "1.8.0"
- python-version: "3.10"
torch: "1.8.1"
- python-version: "3.10"
torch: "1.9.0"
- python-version: "3.10"
torch: "1.9.1"
- python-version: "3.10"
torch: "1.10.0"
- python-version: "3.10"
torch: "1.10.1"
- python-version: "3.10"
torch: "1.10.2"
- python-version: "3.6" # exclude Python 3.6 for [1.11.0]
torch: "1.11.0"
${{ fromJson(needs.generate_build_matrix.outputs.matrix) }}

steps:
# refer to https://github.com/actions/checkout
Expand Down Expand Up @@ -134,25 +107,11 @@ jobs:
./scripts/build_conda_cpu.sh
- name: Display generated files
if: startsWith(matrix.os, 'ubuntu')
run: |
ls -lh /usr/share/miniconda/envs/k2/conda-bld/linux-64
- name: Display generated files
if: startsWith(matrix.os, 'macos')
run: |
ls -lh /usr/local/miniconda/envs/k2/conda-bld/osx-64
- name: Upload generated files
if: startsWith(matrix.os, 'ubuntu')
uses: actions/upload-artifact@v2
with:
name: torch-${{ matrix.torch }}-python-${{ matrix.python-version }}-${{ matrix.os }}
path: /usr/share/miniconda/envs/k2/conda-bld/linux-64/*.tar.bz2

- name: Upload generated files
if: startsWith(matrix.os, 'macos')
uses: actions/upload-artifact@v2
with:
name: torch-${{ matrix.torch }}-python-${{ matrix.python-version }}-${{ matrix.os }}
path: /usr/local/miniconda/envs/k2/conda-bld/osx-64/*.tar.bz2
Original file line number Diff line number Diff line change
Expand Up @@ -15,36 +15,42 @@
# limitations under the License.


name: build-windows-conda
name: build_conda_cpu_windows

on:
push:
branches:
- conda-win
tags:
- '*'

env:
BUILD_TYPE: Release

jobs:
build-windows-conda:
generate_build_matrix:
# see https://github.com/pytorch/pytorch/pull/50633
runs-on: ubuntu-latest
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- name: Generating build matrix
id: set-matrix
run: |
# outputting for debugging purposes
python scripts/github_actions/generate_build_matrix.py
MATRIX=$(python scripts/github_actions/generate_build_matrix.py)
echo "::set-output name=matrix::${MATRIX}"
build_conda_cpu_windows:
# see https://github.com/actions/virtual-environments/blob/win19/20210525.0/images/win/Windows2019-Readme.md
runs-on: ${{ matrix.os }}
needs: generate_build_matrix
runs-on: windows-2019
strategy:
fail-fast: false
matrix:
os: [windows-2019]
# Python 3.9 is for PyTorch 1.7.1, 1.8.x, 1.9.0,
python-version: [3.6, 3.7, 3.8, 3.9]
torch: ["1.5.0", "1.5.1", "1.6.0", "1.7.0", "1.7.1", "1.8.0", "1.8.1", "1.9.0"]
exclude:
- python-version: 3.9 # exclude Python 3.9 for [1.5.0, 1.5.1, 1.6.0, 1.7.0]
torch: "1.5.0"
- python-version: 3.9
torch: "1.5.1"
- python-version: 3.9
torch: "1.6.0"
- python-version: 3.9
torch: "1.7.0"
${{ fromJson(needs.generate_build_matrix.outputs.matrix) }}

steps:
- uses: actions/checkout@v2
Expand Down
Loading

0 comments on commit 7f2ca04

Please sign in to comment.