Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

swig python fixes and added sum fn #175

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

MStumpp
Copy link
Member

@MStumpp MStumpp commented Jan 28, 2018

ToDos/questions:

  • newly created sum fn missing initial value
  • python sum fn in DIA.Sum() and key extractor fn/reduce fn in DIA.ReduceByKey() called with 0 arguments at some point in time during execution causing an exception ("temporarily" fixed by setting default value in python lambda fn)

@bingmann
Copy link
Member

I basically paused working on this, because swig may not be the right wrapper framework.
People have suggested that cython is a better choice.

@bingmann
Copy link
Member

bingmann commented Jan 29, 2018

Furthermore, the elephant lies deeper in this issue: when compiling as a python framework, one wants run the Thrill workers on a distributed system.
This means that one needs python code to which launches on h hosts, and that boots up a Thrill context that connects the embedded framework among the hosts.
This is what I prototyped with the Rpc framework I believe.
However, python probably has better options.
The same then goes for Java.

@MStumpp
Copy link
Member Author

MStumpp commented Jan 29, 2018

So you basically have a tighter integration of Python and Thrill in mind rather than a wrapper-like, loose-coupled one, requiring a 3rd party API (Python, Java, ...) to cold start a Thrill context per host. JNI (rather than JNA) would probably be needed for Java then. Will have a look once time allows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants