Implements hooks support #25

davejlong · 2016-11-18T13:35:31Z

Starting work for #4. The hook API will enable developers to create exs files in hooks/ which will auto load. A hook would look like the following:

hook :slack do
  %{"text" => message} = conn.params
  broadcast! :slack_message, message
end

The slack hook above will generate a route at /hooks/slack. The underlying API of hook/2 will be leveraging the Plug.Route API so hooks will have access to the normal conn object just like any other route.

Remove variable conn is unused compiler warning
Fix issue Implements hooks support #25 (comment)
Have general hook handle params from form, query or JSON
Generate hooks directory when user creates new project

coveralls · 2016-11-18T13:37:10Z

Coverage decreased (-0.4%) to 80.208% when pulling 5a045eb on hooks into 9d3f150 on master.

davejlong · 2016-11-18T13:50:20Z

Would it make sense to, when building the hook allow the dev to pass a custom route to generate?

hook :slack, "/slash" do
  ...
end

Which would then generate /hooks/slash instead of /hooks/slack.

I can't think of a use case, but it's fairly simple to include.

zorbash · 2016-11-19T15:42:59Z

Concerning #25 (comment) i don't think we need such an option right now.

Webhooks are enabled through the `/hooks` route. Building hooks, similar to jobs, is as easy as putting a file in the `hooks/` directory in a Kitto project. The Hook API is very simple and similar to the `Plug.Route` DSL building routes: ```elixir use Kitto.Hooks.DSL hook :github do {:ok, body, _} = read_body conn commits = GitHub.parse_commits_from_hook(body) broadcast! :github_commits, %{commits: commits} end ``` The hook above generates a route at `/hooks/github` listening on all HTTP methods. Hooks include both the `conn` object that routes include as well as `broadcast!/2` to broadcast events to dashboards.

zorbash · 2016-11-22T07:58:50Z

README.md

@@ -132,6 +132,27 @@ end
 The above will spawn a supervised process which will emit a [server-sent
 event](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events) with the name `random` every second.

+## Hooks
+
+When jobs don't work, whether you want something more realtime than polling or


When jobs don't work has a quite negative feeling doesn't it?

zorbash · 2016-11-22T08:00:25Z

lib/kitto/hooks.ex

+
+  defp load_hooks, do: hook_files |> Enum.each(&Code.load_file/1)
+  defp hook_files do
+    [System.cwd, hook_dir, "**/*.{ex,exs}"] |> Path.join |> Path.wildcard


I think you should use Kitto.root instead of System.cwd

zorbash · 2016-11-22T08:01:21Z

lib/kitto/hooks/dsl.ex

+  Hooks act like routes in `Plug.Route` and come complete with the `conn`
+  object for accessing request information.
+  """
+  defmacro __using__(_opts) do


Add @doc false and a newline above.

zorbash · 2016-11-22T08:05:19Z

lib/kitto/hooks/router.ex

+  plug :match
+  plug :dispatch
+
+  match "/:hook" do


Should hooks be authenticated?

I thought about that. I would say no. Instead a lot of APIs when sending webhook requests will add a request signature to the HTTP headers of the request, authenticating that the webhook is coming from where it should be coming from.

We can add authentication later.

String.to_atom hook could be DRYied here.

zorbash · 2016-11-22T08:25:32Z

test/test_helper.exs

@@ -1,6 +1,18 @@
 defmodule Kitto.TestHelper do
  def atomify_map(map) do
-    for {key, value} <- map, into: %{}, do: {String.to_atom(key), value}
+    for {key, value} <- map, into: %{} do
+      {if(is_atom(key), do: key, else: String.to_atom(key)), value}


if is_atom(key), do: ..

That doesn't work in this context. If I change that line to:

{if is_atom(key), do: key, else: String.to_atom(key), value}

It generates a compile time SyntextError: test/test_helper.exs:4: syntax error before: value

zorbash · 2016-11-22T08:27:44Z

lib/kitto/hooks/router.ex

+  plug :dispatch
+
+  match "/:hook" do
+    if Kitto.Hooks.hooks[String.to_atom hook] do


Better alias both Kitto.Hooks and Kitto.Notifier at the top of the module definition.

davejlong · 2016-11-22T14:18:00Z

All changes that could be implemented, or are not up for further discussion, are added.

zorbash · 2016-11-22T15:27:56Z

lib/kitto/hooks/dsl.ex

+  @moduledoc """
+  DSL for building Webhook handlers. Define a new Webhook like follows:
+
+      defmodule MyHook do


You don't have to define a module. It's documented (and tested) differently elsewhere.

zorbash · 2016-11-22T15:29:25Z

test/fixtures/hooks/test_hook.exs

@@ -0,0 +1,3 @@
+use Kitto.Hooks.DSL
+
+hook :hello, do: broadcast! :hello, %{text: "Hello World"}


Tests print a warning about variable conn is unused. We have to do something about it.

Yeah, I saw that. Unfortunately, I cannot make the conn object available in hooks without getting that warning, as not every hook will necessarily use the conn object.

zorbash · 2016-11-22T16:58:52Z

lib/kitto/hooks/router.ex

+defmodule Kitto.Hooks.Router do
+  use Plug.Router
+
+  alias Kitto.{Notifier,Hooks}


[minor] Leave a space after ,.

zorbash · 2016-11-23T11:52:30Z

lib/kitto/hooks.ex

+  defp hook_files do
+    [hook_dir, "**/*.{ex,exs}"] |> Path.join |> Path.wildcard
+  end
+  defp hook_dir, do: Application.get_env(:kitto, :hook_dir, default_hook_dir)


Why would one want to configure the hooks dir?

Part of the reason was to make the logic easier to test, but I thought that maybe someone might want to change it somewhere in the world at some point... maybe.

Let's have it configurable then. I can see how easier it get to test it this way.

zorbash · 2016-11-26T14:18:21Z

@davejlong concerning the unused variable warning, we should go with something like:

defmacro hook(name, do: block) do
  quote do
    # Append the hook to the list of hooks
    Kitto.Hooks.register unquote(name), fn(var!(conn)) ->
       _ = var!(conn)
       unquote(block) 
     end
  end
end

Just like phoenix does at: https://github.com/phoenixframework/phoenix/blob/v1.2/lib/phoenix/endpoint.ex#L376

davejlong · 2016-11-26T15:27:16Z

Agreed. Do you want to go ahead and add it in?

…

On Sat, Nov 26, 2016, 9:18 AM Dimitrios Zorbas ***@***.***> wrote: @davejlong <https://github.com/davejlong> concerning the unused variable warning, we should go with something like: defmacro hook(name, do: block) do quote do # Append the hook to the list of hooks Kitto.Hooks.register unquote(name), fn(var!(conn)) -> _ = var!(conn) unquote(block) end endend Just like phoenix does at: https://github.com/phoenixframework/phoenix/blob/v1.2/lib/phoenix/endpoint.ex#L376 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#25 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKs1egYPqkdqkq-etTa7c_NJzanLkVeks5rCD-ugaJpZM4K2fFq> .

coveralls · 2016-11-28T00:56:43Z

Coverage increased (+1.4%) to 86.486% when pulling fd8aced on hooks into f129784 on master.

davejlong · 2016-11-28T01:04:08Z

I'll work up an example of a Slack slash command using the hooks support, but I think we can go ahead and merge this in. Expect another blog post and PR for the slack command hook later this week.

zorbash · 2016-11-28T18:44:01Z

README.md

+Hooks are stored in the `hooks/` directory and are structured as follows:
+
+```elixir
+# File jobs/github.exs


This should be File: hooks/github.exs.

zorbash · 2016-11-28T18:55:59Z

test/hooks/dsl_test.exs

@@ -0,0 +1,3 @@
+defmodule Kitto.Hooks.DSLTest do


This should go to test/fixtures/hooks/

Whoops. I should actually remove it. It was a test case, but there's nothing in the DSL that isn't being tested elsewhere.

# Conflicts: # lib/kitto.ex

zorbash · 2016-12-04T21:48:39Z

Will review this tomorrow (2016-12-05). It's the last thing keeping 0.3.0 from being released and it looks good.

zorbash · 2016-12-05T18:50:35Z

test/hooks_test.exs

+defmodule Kitto.HooksTest do
+  use ExUnit.Case, async: true
+
+  # setup do


Is this supposed to be commented out?

zorbash · 2016-12-05T18:52:21Z

lib/kitto/hooks.ex

+  def handle_call({:lookup, hook}, _from, hooks) do
+    {:reply, Map.fetch(hooks, hook), hooks}
+  end
+


✂️ Remove newline

zorbash · 2016-12-05T18:53:24Z

lib/kitto/hooks.ex

+defmodule Kitto.Hooks do
+  @moduledoc """
+  Kitto Hooks enable an alternative to Jobs for feeding data into dashboards.
+  Hooks enable remote services to push data into Kitto using webhooks.


Hooks enable remote services to push data into Kitto using webhooks.

This sentence is a bit self-referencing.

zorbash · 2016-12-05T18:53:47Z

lib/kitto/hooks.ex

+  the root of the application. Hooks can be defined as follows:
+
+      use Kitto.Hooks.DSL
+      hook :hello do


I'd leave a newline above the hook definition as we do with jobs.

zorbash · 2016-12-05T18:53:57Z

lib/kitto/hooks.ex

+      use Kitto.Hooks.DSL
+      hook :hello do
+        {:ok, body, _} = read_body conn
+        broadcast! :hello, body |> Poison.decode!


Newline above broadcast!

I also think we should also rewrite broadcast!/2 to broadcast!/1 calls.

zorbash · 2016-12-05T18:56:07Z

README.md

+## Hooks
+
+If, instead of polling for new data from a data source, you want to act on data
+as it changes, hooks are a useful feature for implementing webhooks to feed data


hooks are a useful feature for implementing webhooks

I find this a bit difficult to comprehend. Considering that someone may not be familiar with the concept of hooks, having hooks and webhooks in the same sentence can be a bit confusing.

zorbash · 2016-12-05T18:56:25Z

README.md

+hook :github do
+  {:ok, body, _} = read_body conn
+  commits = GitHub.parse_commits_from_hook(body)
+  broadcast! :github_commits, %{commits: commits}


Add newline above.

zorbash · 2016-12-05T18:57:22Z

README.md

+```
+
+The hook generates a route using the atom in the `hook/2` method. The hook above
+will listen at `/hooks/github` on any HTTP method.


The hook above will respond to any HTTP method but actually only works with requests having a JSON body.

zorbash · 2016-12-05T18:57:49Z

config/test.exs

@@ -5,3 +5,5 @@ config :kitto, templates_dir: "test/fixtures/views"
 config :kitto, default_layout: "layout"

 config :logger, level: :warn
+
+config :kitto, :hook_dir, "test/fixtures/hooks"


Don't we stub this inside the hooks_test?

zorbash · 2016-12-05T18:59:19Z

lib/kitto.ex

@@ -85,6 +86,7 @@ defmodule Kitto do
  defp children(_env) do
    [supervisor(__MODULE__, [], function: :start_server),
     supervisor(Kitto.Notifier, []),
+     worker(Kitto.Hooks, []),


Since it doesn't depend on any of the other children, i think we should start it last.

zorbash · 2016-12-05T19:08:57Z

lib/kitto/hooks/router.ex

@@ -0,0 +1,30 @@
+defmodule Kitto.Hooks.Router do
+  use Plug.Router


I'm surprised credo doesn't nag about the missing @moduledoc here.

zorbash · 2016-12-05T19:11:31Z

lib/kitto/hooks.ex

+  defp hook_files do
+    [hook_dir, "**/*.{ex,exs}"] |> Path.join |> Path.wildcard
+  end
+  defp hook_dir, do: Application.get_env(:kitto, :hook_dir, default_hook_dir)


This should be hooks_dir like https://github.com/kittoframework/kitto/blob/master/lib/kitto/runner.ex#L59

zorbash · 2016-12-05T19:12:45Z

lib/kitto/hooks.ex

+
+  use GenServer
+
+  def start_link do


Starting it like https://github.com/kittoframework/kitto/blob/master/lib/kitto/runner.ex#L17
with a configurable name allows us to test it independently by spawning a new process (see: https://github.com/kittoframework/kitto/blob/master/test/runner_test.exs#L32) and then disposing it.

Making the server name dynamic is doable, but will take some time. I would suggest waiting until 0.4 if this is necessary for releasing the PR.

zorbash · 2016-12-05T19:14:52Z

lib/kitto/hooks.ex

+  ### Callbacks
+
+  def handle_call({:lookup, hook}, from, hooks) when is_atom(hook) do
+    handle_call({:lookup, Atom.to_string(hook)}, from, hooks)


Replace Atom.to_string(hook) with either "#{hook}" or hook |> to_string

zorbash · 2016-12-05T19:16:26Z

lib/kitto/hooks.ex

+  @doc """
+  Registers a new hook into the registry
+  """
+  def register(name, block) do


This should be like https://github.com/kittoframework/kitto/blob/master/lib/kitto/runner.ex#L31

where the first param is a reference to the process.

Is that really necessary since the GenServer is always named :hook_registry?

... and then I saw your previous comment above.

zorbash · 2016-12-05T19:22:14Z

installer/templates/new/hooks/slack.exs

+
+hook :slack do
+  %{"text" => text} = conn.params
+  broadcast! :slack_message, %{text: text}


newline above

zorbash · 2016-12-05T19:26:50Z

installer/templates/new/hooks/slack.exs

+# 3. In the setting for the URL use http://[my.kitto.dashboard]/hooks/slack
+#
+# As with most webhooks, your Kitto API will need to be publicly accessible
+# to Slack. You can build verification using the `token` in the URL parameters


You can build verification using the token in the URL parameters

It's not entirely clear how the user is supposed to build authentication. We should either include it in the example, or mention it elsewhere (wiki).

zorbash · 2016-12-05T19:29:05Z

lib/kitto/hooks.ex

@@ -0,0 +1,68 @@
+defmodule Kitto.Hooks do


Since this module serves as a hook registry why not just name it Kitto.HookRegistry?

zorbash · 2016-12-05T19:29:40Z

lib/kitto/hooks.ex

+
+  def init(:ok) do
+    load_hooks
+    {:ok, %{}}


Add newline above.

coveralls · 2016-12-05T19:50:00Z

Coverage increased (+0.9%) to 87.459% when pulling 8dd0181 on hooks into 55fe04b on master.

coveralls · 2016-12-05T19:55:49Z

Coverage increased (+0.5%) to 87.129% when pulling 7d49e9c on hooks into 55fe04b on master.

davejlong · 2016-12-31T02:34:24Z

I'm going to close this for right now while I rethink the hook system. Will reopen a PR after I revise the concept to be more unified with the job running system.

Starting to implement hook router

5a045eb

davejlong added 3 commits November 21, 2016 09:59

Merge branch 'master' into hooks

40f6717

Merge branch 'master' into hooks

8cce82d

davejlong changed the title ~~[WIP] Implements hooks support~~ Implements hooks support Nov 22, 2016

zorbash reviewed Nov 22, 2016

View reviewed changes

Update docs. Alias Notifier and Hooks in Hook Router

2b58fdd

zorbash reviewed Nov 22, 2016

View reviewed changes

zorbash reviewed Nov 23, 2016

View reviewed changes

davejlong added 2 commits November 27, 2016 19:47

Remove warnings when conn unused by hooks

9573e02

Merge remote-tracking branch 'origin' into hooks

fd8aced

zorbash reviewed Nov 28, 2016

View reviewed changes

davejlong added 2 commits November 28, 2016 14:42

Fix error in README. Removed empty test file

7302037

Merge remote-tracking branch 'origin' into hooks

8e3b97a

# Conflicts: # lib/kitto.ex

zorbash added the kind: enhancement label Dec 4, 2016

zorbash reviewed Dec 5, 2016

View reviewed changes

test/hooks_test.exs

defmodule Kitto.HooksTest do

use ExUnit.Case, async: true

# setup do

Copy link

Member

zorbash Dec 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this supposed to be commented out?

zorbash reviewed Dec 5, 2016

View reviewed changes

zorbash modified the milestones: 0.4.0, 0.3.0 Dec 5, 2016

zorbash reviewed Dec 5, 2016

View reviewed changes

lib/kitto/hooks.ex

def init(:ok) do

load_hooks

{:ok, %{}}

Copy link

Member

zorbash Dec 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add newline above.

Changes to make code more clear and understandable

8dd0181

Remove documentation about authentication from Slack example

7d49e9c

davejlong mentioned this pull request Dec 31, 2016

[WIP] Rewrite registry and DSLs #57

Closed

davejlong closed this Dec 31, 2016

		@@ -0,0 +1,3 @@
		use Kitto.Hooks.DSL

		hook :hello, do: broadcast! :hello, %{text: "Hello World"}

		@@ -0,0 +1,30 @@
		defmodule Kitto.Hooks.Router do
		use Plug.Router

Implements hooks support #25

Implements hooks support #25

Conversation

davejlong commented Nov 18, 2016 • edited Loading

coveralls commented Nov 18, 2016

davejlong commented Nov 18, 2016

zorbash commented Nov 19, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davejlong commented Nov 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zorbash commented Nov 26, 2016

davejlong commented Nov 26, 2016 via email

coveralls commented Nov 28, 2016

davejlong commented Nov 28, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zorbash commented Dec 4, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zorbash Dec 5, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zorbash Dec 5, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Dec 5, 2016

coveralls commented Dec 5, 2016

davejlong commented Dec 31, 2016

davejlong commented Nov 18, 2016 •

edited

Loading

zorbash Dec 5, 2016 •

edited

Loading

zorbash Dec 5, 2016 •

edited

Loading