Skip to content
This repository has been archived by the owner on Feb 2, 2024. It is now read-only.

Add numba_pandas extension #253

Merged
merged 1 commit into from
Nov 3, 2019
Merged

Conversation

PokhodenkoSA
Copy link
Contributor

@PokhodenkoSA PokhodenkoSA commented Oct 24, 2019

This PR shows how to create Numba extension for Pandas according the documentation.

See test numba_pandas/tests/test_pandas.py.

Implicit loading of extension is supported in Numba >= 0.46.
Test file also shows how to load extension explicitly for Numba < 0.46.

For HPAT version not ported to Numba 0.46 it is possible to run the test with Numba 0.46:
comment the line import hpat.compiler in hpat/__init__.py.

This test is not in CI yet.

_init_extension() imports only overloads used in the test. It should be extended for all overloads for Pandas.

I think we should consider moving Pandas overloads to numba_pandas package.

@densmirn densmirn self-requested a review October 25, 2019 07:08
Copy link
Contributor

@shssf shssf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. I think the initialization part should be implemented in Numba. It will require the PR to Numba repository. I don't believe in magic. Numba know nothing about HPAT if regular user use conda install numba. So, we have to implement some code in Numba to pickup HPAT if Pandas used in user code. Particular the same as you do in this PR.
  2. Please look into how many libraries are loaded into memory if just import hpat used in python. I think we need to revise initialization part of the HPAT and load required libraries on demand (not all of them at startup). Perhaps it is better to discuss this offline and maybe in another PR.

numba_pandas/__init__.py Outdated Show resolved Hide resolved
numba_pandas/tests/test_pandas.py Outdated Show resolved Hide resolved
numba_pandas/tests/test_pandas.py Outdated Show resolved Hide resolved
@shssf
Copy link
Contributor

shssf commented Oct 27, 2019

@PokhodenkoSA Do we really need _init_extension in some new file? I think we can implement it in any existed init file

@shssf
Copy link
Contributor

shssf commented Oct 31, 2019

@PokhodenkoSA I would like to merge this PR but we need to move initializer into hpat/__init__ and remove other changes which are not related to setup.py

Remove support numba_python tests for numba<0.46
Removed numba_pandas package and tests. Move extension point to hpat/__init__.py
@pep8speaks
Copy link

Hello @PokhodenkoSA! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

Line 537:1: E124 closing bracket does not match visual indentation

@shssf shssf merged commit 1707927 into IntelPython:master Nov 3, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants