New post consumer component #52

srosset81 · 2018-12-14T09:40:30Z

This component POST received data to the configured URL.
Currently only application/json ContentType is accepted.

The name "Webhook" is maybe not right, I wasn't sure how to call it.

Note: this works fine if the "components chain" is in "push-only", for example if you link a "Post provider" component to this component. However if you add a "NoSQL cache" in between, the Webhook component will not be called and thus the data will not be POSTed to the configured URL.

grizio · 2018-12-22T15:49:40Z

.gitignore

@@ -68,3 +68,6 @@ test/test_result/

 #config
 configuration.js
+
+#IDE-specific
+.idea


Just for your information, you can exclude local-specific files (as IDE or personal directories) in .git/info/exclude file. You can do it for each project or all of the projects of your machine (avoid changing .gitignore of each project you contribute on).

This file already exclude environment-specific files, so do not change here, it is only an information. 🙂

grizio · 2018-12-22T15:52:59Z

static/tag/webhook-editor.tag

+<webhook-editor>
+  <!-- bouton aide -->
+  <div class="contenaireH" style="margin-left:97%">
+    <a href="https://github.com/assemblee-virtuelle/Semantic-Bus/wiki/Composant:-Webhook" target="_blank"><img src="./image/help.png" alt="Aide" width="25px" height="25px"></a>


Aide ? I do not see the bond between aide and webhook.

This is the alt tag over the help icon. I've copied what is done in other .tag files.

grizio · 2018-12-22T15:54:24Z

static/tag/webhook-editor.tag

+    this.updateData = function (dataToUpdate) {
+      this.data = dataToUpdate;
+      this.update();
+    }.bind(this);


To avoid this bind, you can declare functions as: this.updateData = (dataToUpdate) => {…}.

Also, you use two formats:

urlInputChanged(e) {…} // and this.updateData = function (dataToUpdate) {…}.bind(this);

Is it possible to keep only one of them?

Ah I didn't know ES6 arrow functions could be used inside Riot tags.
I've updated the code accordingly.

grizio · 2018-12-22T15:58:44Z

webServices/workSpaceComponentDirectory/webhook.js

+
+  // Function called when another component push to this component
+  pull: function(data, flowData, queryParams) {
+    return new Promise((resolve, reject) => {


You can avoid nested promise construction:

Line 28:

return Promise.reject(new Error('...'));

Line 31+:

return this.fetch(componentData.url, { method: 'POST', body: body, headers: { 'Content-Type': componentData.contentType } }).then(() => ({ data: flowData[0].data }))

Avoiding nested promises reduces memory usage (but not strongly impacting here), enforce composition (.then, .catch) and avoid forgetting cases of termination (here, I do not see one, but if you forgot the .catch, the promise will never end, remains in memory and flood the server).

Great idea, it makes the code much more readable. Thanks !

grizio · 2018-12-22T16:15:09Z

package.json

@@ -65,6 +65,7 @@
    "mongoose": "^5.3.2",
    "mongoose-unique-validator": "^2.0.2",
    "mysql": "^2.13.0",
+    "node-fetch": "^2.3.0",


I believe in this project we use http: https://nodejs.org/api/http.html#http_http_request_options_callback

@simonLouvet : Can you confirm?

@grizio I believe the http/https libraries (like the request library) are fairly low-level. When you deal with more complex situations, for example if you are behind a proxy, if you need to handle CORS (?) or redirections, it will require tweaks. (I've seen that for some components, the follow-redirects library is used just to handle the redirections.)

The advantage of node-fetch is that it pretty much handles all cases.

srosset81 · 2018-12-27T17:06:42Z

Thanks for the review @grizio ! I've made some changes and comments.

srosset81 · 2018-12-27T18:09:01Z

@simonLouvet I've improved the code so that, when the webhook POST fails, it retries after 5s, 25s, 2m, 10m, 50m, 4h and 21h.
For the component name, what do you think of "Repost to URL" ?

grizio

The component name Webhook seems valid to me. Using a name like repost to url will give the technical behavior of the component, not its business function.

grizio · 2018-12-28T15:55:23Z

webServices/workSpaceComponentDirectory/webhook.js

+      return this.fetch(url, options).catch(error => {
+          if( numRetry >= 7 ) {
+              // TODO log the failed webhook posts somewhere ?
+              console.error(error);


How does the promise end in this case?

It ends with no return value.
I don't want to rethrow the error, otherwise Node will stop.

grizio · 2018-12-28T15:56:13Z

webServices/workSpaceComponentDirectory/webhook.js

+              console.log(`Webhook post to ${url} failed, trying again in ${retryInterval}s...`);
+
+              return this.sleep(retryInterval * 1000).then(() => {
+                  this.call_webhook(url, options, numRetry+1);


What is the returning value of the promise in this case?

Right, it should return the result of this.call_webhook. I've made the correction.

grizio · 2018-12-28T15:58:52Z

webServices/workSpaceComponentDirectory/webhook.js

+              // This will retry after 5s, 25s, 2m, 10m, 50m, 4h, 21h
+              const retryInterval = Math.pow(5, numRetry+1);
+
+              console.log(`Webhook post to ${url} failed, trying again in ${retryInterval}s...`);


The log can be considered as an error or at least as a warning. If some endpoint is failing, we will be alerted only 26h hours later.

Yes that's the idea. The URL to call may be down temporarly, IMO this shouldn't be an error.
Do you think we should do it differently ? Should I remove the console.log ?

The log is valid, it is the level I would level up to warning at least.
Maybe not the first times, but if the URL is down 1h (or even 30m), it is already long enough to be informed and maybe to check the issue.

What about this compromise:

if numRetry < 4 (for 10m), console.log (like today)

else if numRetry > 7 (for 21h), console.error and stop (like today)

else (for 10m, 50m and 4h) console.warn (new case)

It could also be a more configurable/smart strategy, I mainly give the idea.

simonLouvet · 2018-12-28T17:30:07Z

@srosset81 name of this component is Post Consumer. @Jacx12 can you create an image for this component. Miror of flow consumer please.

Jacx12 · 2018-12-28T17:44:08Z

it's done https://github.com/assemblee-virtuelle/Semantic-Bus/blob/master/static/image/components/Post_consumer.png

srosset81 · 2019-01-02T10:02:46Z

@simonLouvet @grizio I've renamed to Post consumer and made a few corrections.
For me, it's ready to be merged.

simonLouvet · 2019-01-02T16:11:04Z

Merged. visible in production at the next Docker Push.

New webhook component

6a9a1ce

grizio reviewed Dec 22, 2018

View reviewed changes

srosset81 added 2 commits December 27, 2018 17:43

Use arrows and this. for functions inside the Riot tag

7fda760

Avoid nested promises

e090432

Retry webhook post on failures

2d65e3c

srosset81 requested a review from simonLouvet December 27, 2018 18:09

srosset81 added the enhancement label Dec 27, 2018

grizio reviewed Dec 28, 2018

View reviewed changes

Merge branch 'master' into webhook-component

a8ae055

srosset81 changed the title ~~New webhook component~~ New post consumer component Jan 2, 2019

srosset81 added 3 commits January 2, 2019 10:40

Return the value of call_webhook on retries

1c4f634

Rename Webhook to Post Consumer

1a700c1

Lower case C

3ab4077

simonLouvet merged commit 2c75a75 into assemblee-virtuelle:master Jan 2, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New post consumer component #52

New post consumer component #52

srosset81 commented Dec 14, 2018

grizio Dec 22, 2018

grizio Dec 22, 2018

srosset81 Dec 27, 2018

grizio Dec 22, 2018

srosset81 Dec 27, 2018 •

edited

Loading

grizio Dec 22, 2018

srosset81 Dec 27, 2018 •

edited

Loading

grizio Dec 22, 2018

srosset81 Dec 27, 2018 •

edited

Loading

srosset81 commented Dec 27, 2018

srosset81 commented Dec 27, 2018

grizio left a comment

grizio Dec 28, 2018

srosset81 Jan 2, 2019

grizio Dec 28, 2018

srosset81 Jan 2, 2019 •

edited

Loading

grizio Dec 28, 2018

srosset81 Jan 2, 2019

grizio Jan 2, 2019

simonLouvet commented Dec 28, 2018

Jacx12 commented Dec 28, 2018

srosset81 commented Jan 2, 2019

simonLouvet commented Jan 2, 2019

New post consumer component #52

New post consumer component #52

Conversation

srosset81 commented Dec 14, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srosset81 Dec 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srosset81 Dec 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srosset81 Dec 27, 2018 • edited Loading

Choose a reason for hiding this comment

srosset81 commented Dec 27, 2018

srosset81 commented Dec 27, 2018

grizio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srosset81 Jan 2, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonLouvet commented Dec 28, 2018

Jacx12 commented Dec 28, 2018

srosset81 commented Jan 2, 2019

simonLouvet commented Jan 2, 2019

srosset81 Dec 27, 2018 •

edited

Loading

srosset81 Dec 27, 2018 •

edited

Loading

srosset81 Dec 27, 2018 •

edited

Loading

srosset81 Jan 2, 2019 •

edited

Loading