Skip to content
Kotlin bindings for NumPy
Kotlin C Other
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
buildScr/python change getting absolute path to python native library (fixes #3) Jan 10, 2020
examples update version for kotlin and project Dec 25, 2019
gradle/wrapper Initial commit Dec 2, 2019
src change getting absolute path to python native library (fixes #3) Jan 10, 2020
.gitignore Initial commit Dec 2, 2019
LICENSE add copyright line to license Dec 2, 2019
README.md update version in Readme Jan 10, 2020
build.gradle change getting absolute path to python native library (fixes #3) Jan 10, 2020
gradle.properties
gradlew Initial commit Dec 2, 2019
gradlew.bat Initial commit Dec 2, 2019
settings.gradle Initial commit Dec 2, 2019

README.md

JetBrains incubator project Download GitHub License

Kotlin bindings for NumPy

This project is a Kotlin library, which is a statically typed wrapper for the NumPy library.

Features

  • Statically typed multidimensional arrays
  • Idiomatic API for users with NumPy experience
  • Random, math, linear algebra and other useful functions from NumPy
  • Python allocates memory for arrays and frees memory when JVM GC collects unnecessary arrays
  • Direct access to array data using DirectBuffer
    • Increased performance working with array's data compared to python

Install

Groovy

repositories {
    maven { url "https://kotlin.bintray.com/kotlin-numpy" }
}

dependencies {
    implementation 'org.jetbrains:kotlin-numpy:0.1.2'
}

Kotlin

repositories {
    maven("https://dl.bintray.com/kotlin/kotlin-numpy")
}

dependencies {
    implementation("org.jetbrains:kotlin-numpy:0.1.2")
}

Usage

Kotlin bindings API is very similar to NumPy API. Consider the following programs:

Python

import numpy as np
a = np.arange(15).reshape(3, 5) # ndarray([[ 0,  1,  2,  3,  4],
                                #        [ 5,  6,  7,  8,  9],
                                #        [10, 11, 12, 13, 14]])
print(a.shape == (3, 5))        # True
print(a.ndim == 2)              # True
print(a.dtype.name)             # 'int64'

b = (np.arange(15) ** 2).reshape(3, 5)

c = a * b
print(c)
# [[   0    1    8   27   64]
#  [ 125  216  343  512  729]
#  [1000 1331 1728 2197 2744]]

d = c.transpose().dot(a)
print(d)
# [[10625 11750 12875 14000 15125]
#  [14390 15938 17486 19034 20582]
#  [18995 21074 23153 25232 27311]
#  [24530 27266 30002 32738 35474]
#  [31085 34622 38159 41696 45233]]

Kotlin

import org.jetbrains.numkt.core.*
import org.jetbrains.numkt.math.*
import org.jetbrains.numkt.*

fun main() {
    val a = arange(15).reshape(3, 5) // KtNDArray<Int>([[ 0,  1,  2,  3,  4],
                                                     // [ 5,  6,  7,  8,  9],
                                                     // [10, 11, 12, 13, 14]]

    println(a.shape.contentEquals(intArrayOf(3, 5))) // true
    println(a.ndim == 2)                             // true
    println(a.dtype)                                 // class java.lang.Integer

    // create an array of ints, we square each element and the shape to (3, 5) 
    val b = (arange(15) `**` 2).reshape(3, 5)

    // c is the product of a and b, element-wise
    val c = a * b
    println(c)
    // Output:
    // [[   0    1    8   27   64]
    //  [ 125  216  343  512  729]
    //  [1000 1331 1728 2197 2744]]
    
    // d is the dot product of the transposed c and a
    val d = c.transpose().dot(a)
    println(d)
    // Output:
    // [[10625 11750 12875 14000 15125]
    //  [14390 15938 17486 19034 20582]
    //  [18995 21074 23153 25232 27311]
    //  [24530 27266 30002 32738 35474]
    //  [31085 34622 38159 41696 45233]]

}

Array Creation

Simple ways to create arrays look like this:

    array(arrayOf(1, 2, 3)) // simple flat array: KtNDArray<Int>([1, 2, 3])

    array<Float>(listOf(listOf(15, 13), listOf(2, 31))) // KtNDArray<Float>([[15f, 13f], 
                                                                          // [ 2f, 31f])

    ones<Double>(3, 3, 3) // array of ones. Shape will be (3, 3, 3)

    linspace<Double>( 1, 3, 10 ) // array have 10 numbers from 1 to 3

Basic Operations

Arithmetic operations are supported:

    val a = array(arrayOf(20, 30, 40, 50)) // [20, 30, 40, 50]
    val b = arange(4) // [0, 1, 2, 3]
    val c = a - b // [20 29 38 47]

    b `**` 2 // [0, 1, 4, 9]
    sin(a) * 10 // [ 9.12945251, -9.88031624, 7.4511316, -2.62374854]

Matrix operations:

    val matA = array<Long>(listOf(listOf(1, 1), listOf(0, 1))) // KtNDArray<Long>([[1, 1])
                                                                                // [0, 1]])
    
    val matB = array<Long>(listOf(listOf(2, 0), listOf(3, 4))) // KtNDArray<Long>([[2, 0]
                                                                                // [3, 4]])

    println(matA * matB)
    // elementwise product
    // [[2 0]
    //  [0 4]]

    println(matA `@` matB)
    // matrix product:
    // [[5 4]
    //  [3 4]]

    println(matA.dot(matB))
    // matrix product:
    // [[5 4]
    //  [3 4]]

Override assigment operations, act in place to modify an existing array rather than create a new one.

Note: Be careful when importing operations, kotlin extends the operation. E.g., if a is var, then with a += b, kotlin try to generate code a = a + b. To avoid this, explicitly specify imports.

    val a = ones<Int>(2, 3)
    val b = Random.random(2, 3)
    a *= 3

    b += a

Indexing, Slicing and Iterating

There is indexing with which you can access the item. Also, there is ana analogue of slice python, for skipping use None, e.g. [:8:2] in python, [None..8..2] in kotlin. Iteration occurs elementwise regardless of shape.

    val a = arange(10L) `**` 3 // KtNDArray<Long>([0, 1, 8, 27, 64, 125 216 343 512 729])

    //  a[2]
    println(a[2]) // 8

    println(a[2..5..1]) // equivalent a[2:5] in python. output: KtNDArray<Long>([8, 27, 64])

    // equivalent to a[0:6:2] = -1000 in python; from start to position 6, set every 2nd element to -1000
    a[0..6..2] = -1000
    println(a) // [-1000, 1, -1000, 27, -1000, 125, 216, 343, 512, 729]

    // reverse
    println(a[None..None..-1]) // [729, 512, 343, 216, 125, -1000, 27, -1000, 1, -1000]

    for (el in a.reshape(2, 5)) {
        print("${el.toDouble().pow(1.0 / 3.0)} ") // NaN 1.0 NaN 3.0 4.999999999999999 5.999999999999999 ...
    }

It is important to consider that when indexing, you must pass the number of indexes equal to the dimension of the KtNDArray (| j1, j2, j3, ... | = ndim). When slicing, this is not necessary.

    val x = array(arrayOf(1, 2, 3, 4, 5, 6)).apply { resize(2, 3, 1) }
    println(x.shape.joinToString())
//    2, 3, 1

    println(x[1..2])
//    [[[4]
//      [5]
//      [6]]]
      
    println(x[1, 2, 0])
//    6

Stacking

    val a = floor(10 * Random.random(2, 2))
    val b = floor(10 * Random.random(2, 2))

    // stack arrays row wise
    val v = vstack(a, b)
    println(v.shape.joinToString())
    // 4, 2

    // stack arrays column wise
    val h = hstack(a, b)
    println(h.shape.joinToString())
    // 2, 4

NumPy routine coverage

  • Array creation
    • Ones and zeros -
    • From existing data -
    • Creating record arrays - ⬜️
    • Numeric character arrays - ⬜️
    • Numerical ranges -
    • Building matrices -
    • The Matrix class -
  • Array manipulation routines
    • Basic operations -
    • Changing array shape -
    • Transpose-like operations -
    • Changing number of dimensions -
    • Changing kind of array -
    • Joining arrays -
    • Splitting arrays -
    • Tiling arrays -
    • Adding and removing elements -
    • rearranging elements -
  • Binary operations
    • Elementwise bit operations -
    • Bit packing -
    • Output formatting -
  • String operations - ⬜️
  • C-Types Foreign Function Interface (ctypeslib) - ⬜️
  • Datetime Support Functions - ⬜️
  • Data type routines - ⬜️
  • Optionally Scipy-accelerated routines - ⬜️
  • Mathematical functions with automatic domain (numpy.emath) - ⬜️
  • Floating point error handling - ⬜️
  • Discrete Fourier Transform (numpy.fit) - ⬜️
  • Financial functions - ⬜️
  • Functional programming - ⬜️
  • NumPy-specific help functions - ⬜️
  • Indexing routines
    • Generating index arrays - ⬜️
    • Indexing-like operations -
    • Inserting data into arrays - ⬜️
    • Iterating over arrays - ⬜️
  • Input and output
    • NumPy binary files (NPY, NPZ) - ⬜️
    • Text files -
    • Raw binary files -
    • String formatting - ⬜️
    • Memory mapping files - ⬜️
    • Text formatting options - ⬜️
    • Base-n representations - ⬜️
    • Data sources - ⬜️
    • Binary format description - ⬜️
  • Linear algebra (numpy.linalg)
    • Matrix and vector products -
    • Decompositions -
    • Matrix eigenvalues -
    • Norms and other numbers -
    • Solving equations and inverting matrices - without tensors
  • Logic functions
    • Truth value testing -
    • Array contents -
    • Array type testing - ⬜️
    • Logic operations -
    • Comparison -
  • Masked array operations - ⬜️
  • Mathematical functions
    • Trigonometric functions -
    • Hyperbolic functions -
    • Rounding -
    • Sums, products, differences -
    • Exponents and logarithms -
    • Other special functions -
    • Floating point routines -
    • Rational routines -
    • Arithmetic operations -
    • Handling complex numbers - ⬜️
    • Miscellaneous -
  • Matrix library (numpy.matlib) - ⬜️
  • Miscellaneous routines - ⬜️
  • Padding arrays - ⬜️
  • Polynomials - ⬜️
  • Random (numpy.random)
    • Simple random data -
    • Permutations -
    • Distributions -
    • Random generator - ⬜️
  • Set routines - ⬜️
  • Sorting, searching and counting
    • Sorting -
    • Searching -
    • Counting -
  • Statistics
    • Order statistics -
    • Averages and variances -
    • Correlating -
    • Histograms - ⬜️
  • Test support - ⬜️
  • Window functions - ⬜️

How it works

Foundation

Using Java Native Interface (JNI) and Python C Extensions, we attach the python interpreter to the JVM process. There is a singleton Interpreter for this. Initialization of the Python interpreter occurs on the first call of any function. The interpreter will remain in the JVM until the JVM exits. The Interpreter class contains external functions (as callFunc) through which NumPy functions are called. Consider the call to NumPy functions in more detail.

The following arguments are required to call NumPy functions:

  • Array of call attributes - module names and function names in NumPy
  • Array of arguments - function arguments, equivalent to *args in Python. Importantly, the arguments should be in the same order and in the same places as in Python. If some argument is not passed, None is used instead.
  • Map of arguments - maps name of arguments to arguments, equivalent to **kwargs in Python.
  • Return type - which class we expect to return. If the expected type is KtNDArray, then this is not needed.

To call the NumPy function, the above arguments are passed to the associated external kotlin function. After that, the following code will appear in native code:

  • By the array of call attributes we get callable PyObject. This is a related function NumPy.
  • The array of arguments is converted to a tuple of the corresponding Python objects.
  • If the map of argument is not empty, then it is converted into a dictionary of keyword arguments.
  • We call a NumPy function with arguments passed to it - PyObject_Call(FunctionObject, TupleArgs, DictKwargs), in python it is FunctionObject(TupleArgs, DictKwargs)
  • Convert the result in Java type and returned.

Let's look at an example. The diagonal method returns the specified diagonal, of the same type as the called KtNDArray object.

fun <T : Any> KtNDArray<T>.diagonal(offset: Int = 0, axis1: Int = 0, axis2: Int = 1): KtNDArray<T> =
    callFunc(nameMethod = arrayOf("ndarray", "diagonal"), args = arrayOf(this, offset, axis1, axis2))

We see that the first argument is an array of strings. Getting the final attribute will look like this: numpy -> ndarray -> diagonal (the numpy module is used by default). The next argument is an array of arguments: this (KtNDArray from which the diagonals are taken), offset, axis1, axis2. In this case kwargs are note used. We also expect the array to return, so we don’t pass anything to the return type.

Objects

Type Matching

When processing objects in native code, there is a conversion from Java objects to Python objects, and when the result is returned back - from Python Objects to Java objects. The table below shows the conversion of objects of different types.

Kotlin -> Python Python/NumPy -> Kotlin
None -> None None -> null
Char -> str
String -> str str -> String
Boolean -> bool bool -> Boolean
Byte -> int8 int8 -> Byte
Short -> int16 int16 -> Short
Int -> int32 int32 -> Int
Long -> int64 int64 -> Long
Float -> float32 float32 -> Float
Double -> float64 float64 -> Double
Array -> tuple tuple -> Array
List -> list list -> List
Map -> dict dict -> Map
KtNDArray -> ndarray ndarray -> KtNDArray
Slice -> slice

What's inside KtNDArray

The main object is KtNDArray, like ndarray in NumPy, it is a homogeneous multidimensional array. KtNDArray holds a pointer to its corresponding ndarray. Using the pointer, we can perform operations on the array.

KtNDArray and ndarray operate on shared memory. Python allocates memory for the array and through java.nio.DirectByteBuffer we get access to this memory.

KtNDArray provides access to some ndarray attributes, such as shape, ndim, itemsize, size, strides, dtype. Also KtNDArray has data field of type ByteBuffer. This is the direct buffer.

Type Safety

Kotlin is a statically typed programing language. This makes it possible to catch errors at the compilation stage.

Lest's look at an example:

import numpy as np

# ...

a = np.ones((3, 3), dtype=int) * 3
b = np.random.random((3, 3))

b *= a # successfully

a *= b # get at runtime TypeError

The same code on Kotlin will notify us of an error during compilation:

// ...

val a = ones<Int>(3, 3) * 3
val b = Random.random(3, 3)

b *= a // successfully

a *= b // Get compilation error
// Kotlin: Type mismatch: inferred type is KtNDArray<Double> but KtNDArray<Int> was expected

A few more errors that can be prevented in the kotlin at the compilation stage

When passing parameters to functions: Python

a = np.array(['a', 'b', 'c'])
np.sin(a)

# TypeError: ufunc 'sin' not supported for the input types, 
# and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Kotlin

val a = array(arrayOf('a', 'b', 'c'))
sin(a)

// Kotlin: Type parameter bound for T in fun <T : Number> sin(x: KtNDArray<T>): KtNDArray<Double> 
// is not satisfied: inferred type Char is not a subtype of Number

Method signature defined:

Python

a = np.array([0, 1], [1, 0])

# TypeError: data type not understood

Kotlin

val a = array(listOf(0, 1), listOf(1, 0))

// Kotlin: None of the following functions can be called with the arguments supplied: ...

Implicit type conversions will make you look for errors for a long time. The following code will be executed, but implicitly converts float -> int, which the user may not expect and as a result, the final result will differ from the desired

Python

a = np.arange(15, dtype=np.int32)
b = np.linspace(0, 1, 15, dtype=np.float64)[::-1]
for i in range(15):
    a[i] = b[i]
print(a)

# [1 0 0 0 0 0 0 0 0 0 0 0 0 0 0]

In Kotlin, the compiler will tell the user that an explicit conversion is required here.

Kotlin

val a = arange(15)
val b = linspace<Double>(0, 1, 15)[None..None..-1]
for (i in 0..14) {
    a[i] = b[i] // Error
}

Requirements

To run library you need:

  • Java >= 8
  • Python >= 3.5
  • NumPy >= 1.7

Note: Make sure you use the correct python environment. This is necessary to use the correct version of Python and NumPy.

For the convenience of installing Python, NumPy and setting the environment, it's recommended to use Anaconda.

Building

The build you need Gcc or Clang.

This library is built with Gradle. You must first build the native library: run ./gradlew ktnumpyReleaseSharedLibrary. The library will be in ./build/libs/ktnumpy/shared/release.

Run for debug: ./gradlew ktnumpyDebugSharedLibrary. Accordingly, the library will be in ./build/libs/ktnumpy/shared/debug.

After building the native library, to run ./gradlew build

You can’t perform that action at this time.