New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for custom functions #68

Closed
pauldix opened this Issue Nov 17, 2013 · 35 comments

Comments

Projects
None yet
@pauldix
Member

pauldix commented Nov 17, 2013

User should be able to define custom functions written in Lua like in Redis. This will require the addition of a few new endpoints:

# add a function
curl -X POST http://localhost:8086/db/foo/functions -d \
 '{"name": "square", "arg_count": 1, "map": "...", "reduce": "..."}'

# list functions
curl http://localhost:8086/db/foo/functions

# remove function
curl -X DELETE http://localhost:8086/db/foo/functions/:name

# update function
curl -X PUT http://localhost:8086/db/foo/functions/:name -d '{...}'

The name must be alphanumeric or with underscores. The arg count lets us know how many arguments the function takes. The map and reduce arguments are the Lua scripts to run the associated phases of the query.

We'll have to figure out the most efficient way to marshal and unmarshal the data between Go and Lua. That will inform what the underlying API will look like in Lua for writing these scripts.

@FGRibreau

This comment has been minimized.

Show comment
Hide comment
@FGRibreau

FGRibreau Nov 18, 2013

Just my two cents: I don't know about others but IMHO this feature does not seem really critical compared to the ability of limiting memory usage and the SQL alias support.

FGRibreau commented Nov 18, 2013

Just my two cents: I don't know about others but IMHO this feature does not seem really critical compared to the ability of limiting memory usage and the SQL alias support.

@pauldix

This comment has been minimized.

Show comment
Hide comment
@pauldix

pauldix Nov 18, 2013

Member

Those are definitely higher priority. Just putting this up to track things
and so other people know where we're going. Maybe we should set a few
milestones and assign to those so people can get a sense of the priority of
things?

On Mon, Nov 18, 2013 at 4:13 AM, Francois-Guillaume Ribreau <
notifications@github.com> wrote:

Just my two cents: I don't know about others but IMHO this feature does
not seem really critical compared to the ability of limiting memory usage
and the SQL alias support.


Reply to this email directly or view it on GitHubhttps://github.com//issues/68#issuecomment-28683572
.

Member

pauldix commented Nov 18, 2013

Those are definitely higher priority. Just putting this up to track things
and so other people know where we're going. Maybe we should set a few
milestones and assign to those so people can get a sense of the priority of
things?

On Mon, Nov 18, 2013 at 4:13 AM, Francois-Guillaume Ribreau <
notifications@github.com> wrote:

Just my two cents: I don't know about others but IMHO this feature does
not seem really critical compared to the ability of limiting memory usage
and the SQL alias support.


Reply to this email directly or view it on GitHubhttps://github.com//issues/68#issuecomment-28683572
.

@obfuscurity

This comment has been minimized.

Show comment
Hide comment
@obfuscurity

obfuscurity Nov 19, 2013

I'm not in the business of prioritizing others' work without actually submitting code, so I'll only say that your efforts are appreciated either way. All of these things are useful. 👍

obfuscurity commented Nov 19, 2013

I'm not in the business of prioritizing others' work without actually submitting code, so I'll only say that your efforts are appreciated either way. All of these things are useful. 👍

@FGRibreau

This comment has been minimized.

Show comment
Hide comment
@FGRibreau

FGRibreau Nov 19, 2013

@obfuscurity that was just an unpretentious feedback from an early adopter, gosh, you are rude.
@pauldix yes, milestone would be awesome to track down what is going on :)

FGRibreau commented Nov 19, 2013

@obfuscurity that was just an unpretentious feedback from an early adopter, gosh, you are rude.
@pauldix yes, milestone would be awesome to track down what is going on :)

@obfuscurity

This comment has been minimized.

Show comment
Hide comment
@obfuscurity

obfuscurity Nov 19, 2013

I totally am the rude one, prioritizing other folks' open source efforts. /s

obfuscurity commented Nov 19, 2013

I totally am the rude one, prioritizing other folks' open source efforts. /s

@pauldix

This comment has been minimized.

Show comment
Hide comment
@pauldix

pauldix Nov 19, 2013

Member

Ok, we're all friends here. It's all good and I think the milestone thing is a fine idea. We definitely want more feedback than less and it helps to hear what features people think are the most important.

Member

pauldix commented Nov 19, 2013

Ok, we're all friends here. It's all good and I think the milestone thing is a fine idea. We definitely want more feedback than less and it helps to hear what features people think are the most important.

@flier

This comment has been minimized.

Show comment
Hide comment
@flier

flier May 28, 2014

I'm working on a prototype to support custom functions, just like

CREATE FUNCTION foo(bar) 
begin
  // lua scripts
end

and plan to support use those functions as aggregator at least, like

select foo(bytes) from flow

are there anybody has interesting to contribute code or your feedback?

https://github.com/flier/influxdb/tree/influxdb-lua/src

The current implementation can parse CREATE FUNCTION and I could finish shard's lua engine this week

flier commented May 28, 2014

I'm working on a prototype to support custom functions, just like

CREATE FUNCTION foo(bar) 
begin
  // lua scripts
end

and plan to support use those functions as aggregator at least, like

select foo(bytes) from flow

are there anybody has interesting to contribute code or your feedback?

https://github.com/flier/influxdb/tree/influxdb-lua/src

The current implementation can parse CREATE FUNCTION and I could finish shard's lua engine this week

@pauldix

This comment has been minimized.

Show comment
Hide comment
@pauldix

pauldix May 28, 2014

Member

Very cool! I have a few comments that might help.

I don't know that the custom function definition needs to be in the query
language. It could just be a couple of endpoints in the HTTP api:
Create/update (PUT), show, list, delete. Might make things easier.

I have a few ideas on how it should work overall. The function definitions
should be replicated via Raft and kept in the cluster configuration. The
config.toml file should have options for enforcing limits on the lua
runtime (CPU, Mem). Or this could possibly be cluster wide and replicated
via Raft.

The shard isn't really the right level to implement this on. It should
exist within the engine code. That way the coordinator logic that either
executes the engine locally (thus giving you data locality) or all on a
single server should just work.

The lua runtime that is running the function should be able to keep some
sort of state. The new defined function will almost always receive streams
of points. So it'll have to keep state between each yield of points to the
custom function.

It would be nice within the lua part of the custom function if there were
methods to query InfluxDB or write out points. Maybe this is something that
could be added later.

Just a brain dump of some things I was thinking on this.

On Tue, May 27, 2014 at 11:29 PM, Flier Lu notifications@github.com wrote:

I'm working on a prototype to support custom functions, just like

CREATE FUNCTION foo(bar)
begin
// lua scripts
end

and plan to support use those functions as aggregator at least, like

select foo(bytes) from flow

are there anybody has interesting to contribute code or your feedback?

https://github.com/flier/influxdb/tree/influxdb-lua/src

The current implementation can parse CREATE FUNCTION and I could finish
shard's lua engine this week


Reply to this email directly or view it on GitHubhttps://github.com//issues/68#issuecomment-44362131
.

Member

pauldix commented May 28, 2014

Very cool! I have a few comments that might help.

I don't know that the custom function definition needs to be in the query
language. It could just be a couple of endpoints in the HTTP api:
Create/update (PUT), show, list, delete. Might make things easier.

I have a few ideas on how it should work overall. The function definitions
should be replicated via Raft and kept in the cluster configuration. The
config.toml file should have options for enforcing limits on the lua
runtime (CPU, Mem). Or this could possibly be cluster wide and replicated
via Raft.

The shard isn't really the right level to implement this on. It should
exist within the engine code. That way the coordinator logic that either
executes the engine locally (thus giving you data locality) or all on a
single server should just work.

The lua runtime that is running the function should be able to keep some
sort of state. The new defined function will almost always receive streams
of points. So it'll have to keep state between each yield of points to the
custom function.

It would be nice within the lua part of the custom function if there were
methods to query InfluxDB or write out points. Maybe this is something that
could be added later.

Just a brain dump of some things I was thinking on this.

On Tue, May 27, 2014 at 11:29 PM, Flier Lu notifications@github.com wrote:

I'm working on a prototype to support custom functions, just like

CREATE FUNCTION foo(bar)
begin
// lua scripts
end

and plan to support use those functions as aggregator at least, like

select foo(bytes) from flow

are there anybody has interesting to contribute code or your feedback?

https://github.com/flier/influxdb/tree/influxdb-lua/src

The current implementation can parse CREATE FUNCTION and I could finish
shard's lua engine this week


Reply to this email directly or view it on GitHubhttps://github.com//issues/68#issuecomment-44362131
.

jvshahid pushed a commit that referenced this issue Aug 12, 2014

@jvshahid jvshahid removed the enhancement label Oct 9, 2014

@tmaiaroto

This comment has been minimized.

Show comment
Hide comment
@tmaiaroto

tmaiaroto Oct 15, 2014

+1 is about all I can really say here. I just wanted someone to know this is a feature at least someone else (me) would very much like to see.

tmaiaroto commented Oct 15, 2014

+1 is about all I can really say here. I just wanted someone to know this is a feature at least someone else (me) would very much like to see.

@toddboom toddboom added the idea label Oct 15, 2014

@YaronWittenstein

This comment has been minimized.

Show comment
Hide comment
@YaronWittenstein

YaronWittenstein Oct 16, 2014

Will it be possible to hook into series insertion event such that we could decide to exclude persisting of points?
might be very useful for omitting noises from series

YaronWittenstein commented Oct 16, 2014

Will it be possible to hook into series insertion event such that we could decide to exclude persisting of points?
might be very useful for omitting noises from series

@trink

This comment has been minimized.

Show comment
Hide comment
@trink

trink Oct 24, 2014

@pauldix I believe the Lua sandbox we built for Heka matches your needs and is pretty easy to define your own API on top of https://github.com/mozilla-services/lua_sandbox/blob/dev/docs/sandbox_api.md

trink commented Oct 24, 2014

@pauldix I believe the Lua sandbox we built for Heka matches your needs and is pretty easy to define your own API on top of https://github.com/mozilla-services/lua_sandbox/blob/dev/docs/sandbox_api.md

@killerwolf

This comment has been minimized.

Show comment
Hide comment
@killerwolf

killerwolf commented Nov 13, 2014

+1

1 similar comment
@bbinet

This comment has been minimized.

Show comment
Hide comment
@bbinet

bbinet Dec 11, 2014

Contributor

+1

Contributor

bbinet commented Dec 11, 2014

+1

@aviau

This comment has been minimized.

Show comment
Hide comment
@aviau

aviau Dec 29, 2014

Member

+1 - Nice feature

Member

aviau commented Dec 29, 2014

+1 - Nice feature

@majj

This comment has been minimized.

Show comment
Hide comment
@majj

majj commented Jan 4, 2015

+1

7 similar comments
@allertec

This comment has been minimized.

Show comment
Hide comment
@allertec

allertec commented Jan 19, 2015

+1

@madazone

This comment has been minimized.

Show comment
Hide comment
@madazone

madazone commented Jan 21, 2015

+1

@lxcid

This comment has been minimized.

Show comment
Hide comment
@lxcid

lxcid commented Jan 22, 2015

+1

@inthecloud247

This comment has been minimized.

Show comment
Hide comment
@inthecloud247

inthecloud247 commented Feb 17, 2015

+1

@tobiemh

This comment has been minimized.

Show comment
Hide comment
@tobiemh

tobiemh commented Feb 18, 2015

+1

@thancock14

This comment has been minimized.

Show comment
Hide comment
@thancock14

thancock14 commented Apr 2, 2015

+1

@arnoldbechtoldt

This comment has been minimized.

Show comment
Hide comment
@arnoldbechtoldt

arnoldbechtoldt commented Apr 4, 2015

+1

@beckettsean beckettsean added RFC and removed idea labels Apr 8, 2015

@beckettsean beckettsean added this to the Longer term milestone Apr 8, 2015

@stephanbuys

This comment has been minimized.

Show comment
Hide comment
@stephanbuys

stephanbuys commented May 13, 2015

+1

@srini-raju

This comment has been minimized.

Show comment
Hide comment
@srini-raju

srini-raju commented Jun 18, 2015

+1

1 similar comment
@ruiqi

This comment has been minimized.

Show comment
Hide comment
@ruiqi

ruiqi commented Jul 8, 2015

+1

@nathholf

This comment has been minimized.

Show comment
Hide comment
@nathholf

nathholf commented Oct 8, 2015

+1

4 similar comments
@TheFlyingCorpse

This comment has been minimized.

Show comment
Hide comment
@TheFlyingCorpse

TheFlyingCorpse commented Oct 10, 2015

+1

@cloud-rocket

This comment has been minimized.

Show comment
Hide comment
@cloud-rocket

cloud-rocket commented Oct 22, 2015

+1

@timgriffiths

This comment has been minimized.

Show comment
Hide comment
@timgriffiths

timgriffiths commented Nov 2, 2015

+1

@yuankui

This comment has been minimized.

Show comment
Hide comment
@yuankui

yuankui commented Nov 4, 2015

+1

@padcom

This comment has been minimized.

Show comment
Hide comment
@padcom

padcom Dec 14, 2015

I have a case where the value of pressure stored in the database is an absolute value as retrieved from the sensor. It'd be a perfect case for a custom function to be able to translate that value to sea-level.

+1

padcom commented Dec 14, 2015

I have a case where the value of pressure stored in the database is an absolute value as retrieved from the sensor. It'd be a perfect case for a custom function to be able to translate that value to sea-level.

+1

@toddboom

This comment has been minimized.

Show comment
Hide comment
@toddboom

toddboom Jan 21, 2016

Contributor

Support for UDFs (User-Defined Functions) will be released within Kapacitor for the v0.10.0 release in the next couple weeks. It's unlikely that InfluxDB itself will ever re-implement this functionality, so I'm closing this out.

Contributor

toddboom commented Jan 21, 2016

Support for UDFs (User-Defined Functions) will be released within Kapacitor for the v0.10.0 release in the next couple weeks. It's unlikely that InfluxDB itself will ever re-implement this functionality, so I'm closing this out.

@toddboom toddboom closed this Jan 21, 2016

@yuankui

This comment has been minimized.

Show comment
Hide comment
@yuankui

yuankui Jan 27, 2016

I am forced to use influxdata family...
though I don't think UDFs in Kapacitor will solve the performance problem

yuankui commented Jan 27, 2016

I am forced to use influxdata family...
though I don't think UDFs in Kapacitor will solve the performance problem

@cxreg

This comment has been minimized.

Show comment
Hide comment
@cxreg

cxreg Feb 19, 2016

this strikes me as a particularly odd "resolution". kapacitor is just a client of influxdb, correct?

cxreg commented Feb 19, 2016

this strikes me as a particularly odd "resolution". kapacitor is just a client of influxdb, correct?

@mitar

This comment has been minimized.

Show comment
Hide comment
@mitar

mitar Jun 22, 2016

The discussion moved here: #6891

Or maybe this issue should just be reopened? It is really strange. It seems this is one of the most wanted features? And it is not even on a roadmap? @beckettsean? @pauldix?

mitar commented Jun 22, 2016

The discussion moved here: #6891

Or maybe this issue should just be reopened? It is really strange. It seems this is one of the most wanted features? And it is not even on a roadmap? @beckettsean? @pauldix?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment