Support fetch external resources over HTTP by xddxdd · Pull Request #1007 · DNSControl/dnscontrol

xddxdd · 2020-12-26T17:24:18Z

This PR adds Fetch API to dnscontrol, mentioned in issue #994, via this extension for otto https://github.com/deoxxa/ottoext.

In addition, a PANIC(reason) function is added to abort dnscontrol execution with an exit code, in case an error happened.

And a FETCH() wrapper function is added to automatically call PANIC() if a network error occurred.

Fixes #994

Example: successful request

dnscontrol.js:

var REG_NONE = NewRegistrar('none', 'NONE');
var DNS_BIND = NewDnsProvider('bind', 'BIND');

D('example.com', REG_NONE, DnsProvider(DNS_BIND), [
  A('@', '1.2.3.4'),
]);

console.log('Hello World');
FETCH('https://example.com').then(function(r) {
  return r.text();
}).then(function(t) {
  // Do something with response body
  D_EXTEND('example.com', [
    TXT('@', t.slice(0, 100)),
  ]);
});
console.log('Goodbye');

Generated zone:

$TTL 300
; generated with dnscontrol 2020-12-27T00:57:20+08:00
@                IN SOA   DEFAULT_NOT_SET. DEFAULT_NOT_SET. 2020122601 3600 600 604800 1440
                 IN A     1.2.3.4
                 IN TXT   "<!doctype html>\010<html>\010<head>\010    <title>Example Domain</title>\010\010    <meta charset=\"utf-8\" />\010    <m"

Example: failed request

dnscontrol.js:

var REG_NONE = NewRegistrar('none', 'NONE');
var DNS_BIND = NewDnsProvider('bind', 'BIND');

D('example.com', REG_NONE, DnsProvider(DNS_BIND), [
  A('@', '1.2.3.4'),
]);

console.log('Hello World');
// Example of failure: invalid URL
FETCH('httttttps://example.com').then(function(r) {
  return r.text();
}).then(function(t) {
  // Do something with response body
  D_EXTEND('example.com', [
    TXT('@', t.slice(0, 100)),
  ]);
});
console.log('Goodbye');

Dnscontrol output:

Hello World
Goodbye
Error: Get "httttttps://example.com": unsupported protocol scheme "httttttps"

xddxdd · 2020-12-26T17:33:38Z

I'm unsure if I should add unit tests for the fetch functionality, since I didn't find individual tests for basic functions like A(). Please let me know if a unit test for this is necessary.

tlimoncelli

A. It is fabulous that you can add such a big feature in so few lines of code. Good job!

I have 2 concerns:

There have been no updates to fknsrs.biz/p/ottoext in years. Is it abandoned or just very stable?
This will encourage people to make dnsconfig.js configs that break when a third-party dependency is broken or down. I think that the docs should include a warning against complexity.

B. Please add docs (docs/_functions/global/FETCH.md)

C. Unit tests would be great:

I'm unsure if I should add unit tests for the fetch functionality, since I didn't find individual tests for basic functions like A(). Please let me know if a unit test for this is necessary.

My knowledge of Javascript is limited. If you can help me make unit tests, I would appreciate it!

tlimoncelli · 2020-12-28T21:10:42Z

 	currentDirectory = filepath.Dir(file)

 	vm := otto.New()
+	l := loop.New(vm)


Please add comments explaining these timers.

xddxdd · 2021-01-01T08:14:22Z

There have been no updates to fknsrs.biz/p/ottoext in years. Is it abandoned or just very stable?

The project seems abandoned, but from the testing the functionalities are stable. Nevertheless I can take over and maintain it.

For example, I just forked that project and added the functionality to specify request headers, and updated this PR to point to my fork. Please let me know if it's acceptable to you.

Please add docs (docs/_functions/global/FETCH.md)

I've added docs for the two functions I added, FETCH and PANIC.

This will encourage people to make dnsconfig.js configs that break when a third-party dependency is broken or down. I think that the docs should include a warning against complexity.

A warning is included in the docs for FETCH function.

Unit tests would be great:

On a second thought, the ottoext project already includes its own unit tests, so I doubt if adding separate unit tests for FETCH is necessary. If needed, I can reuse the unit tests from ottoext.

tlimoncelli · 2021-01-04T15:11:59Z

Yes, using your fork is good.

I don't see the new docs. Did you "git add" them? (or maybe it's a problem on my side?)

xddxdd · 2021-01-04T15:15:15Z

Oh wait, I forgot to push the doc changes to remote.

Should be fixed now.

xddxdd · 2021-01-04T16:17:26Z

Speaking of security, I just realized that a switch is needed to explicitly enable this functionality, along with some additional warnings.

Or, in the case where DnsControl runs in a CI, based on data from a git repo that accepts and tests PRs, I can easily turn the CI into a DDoS machine with some while true do fetch(victim) thing.

tlimoncelli · 2021-01-04T16:19:30Z

Good point. It would be good to enable this only with a flag.

add PANIC() and error-handled FETCH()

xddxdd · 2021-01-05T17:32:41Z

A flag --allow-fetch is added to enable fetch. Otherwise dnscontrol will exit with the following error if fetch is used:

executing dnsconfig.js: ReferenceError: 'fetch' is not defined

tlimoncelli

LGTM. Ready for merge?

xddxdd · 2021-01-06T05:10:53Z

Sure, good to go.

tlimoncelli · 2021-01-06T15:49:02Z

Thanks!

P.S. I'd love to hear about how people use this feature!

patschi · 2021-01-08T21:14:08Z

Sorry for being late to the party: A usecase I can think of is querying the DKIM public key from my mailserver setup (using mailcow) from the API and add the records to the DNS zone. However this requires authenticating on the API via POST parameters, but when I see it correct the current fetch() implementation isn't capable of POST'ing values on the same request?

xddxdd · 2021-01-09T08:53:46Z

@patschi Technically POSTing parameters is supported, but for now you'll have to handcraft the request body. For example, both snippers below POST "a=12345" and "b=54321".

This is the easier way of form-urlencoded, useful to pass a pair of API key/secret along:

FETCH('http://localhost/test.php', {
  headers: {
    "Content-Type": "application/x-www-form-urlencoded",
  },
  method: "POST",
  body: 'a=12345&b=54321',
}).then(function(r) {
  return r.text();
}).then(function(t) {
  console.log(t);
});

This is the more complicated way of creating a multipart form-data.

data = '--------------------------2c3e2cab183d59b3' + "\n"
     + 'Content-Disposition: form-data; name="a"' + "\n"
     + '' + "\n"
     + '12345' + "\n"
     + '--------------------------2c3e2cab183d59b3' + "\n"
     + 'Content-Disposition: form-data; name="b"' + "\n"
     + '' + "\n"
     + '54321' + "\n"
     + '--------------------------2c3e2cab183d59b3--' + "\n";

FETCH('http://localhost/test.php', {
  headers: {
    "Content-Type": "multipart/form-data; boundary=------------------------2c3e2cab183d59b3",
  },
  method: "POST",
  body: data,
}).then(function(r) {
  return r.text();
}).then(function(t) {
  console.log(t);
});

tlimoncelli reviewed Dec 28, 2020

View reviewed changes

xddxdd force-pushed the pr-fetch branch 2 times, most recently from 9d592d8 to 8b30a48 Compare January 1, 2021 07:49

xddxdd force-pushed the pr-fetch branch from 8b30a48 to 1d0e73c Compare January 4, 2021 15:14

tlimoncelli reviewed Jan 4, 2021

View reviewed changes

Comment thread pkg/js/js.go Outdated

xddxdd force-pushed the pr-fetch branch from 1d0e73c to 1ba9808 Compare January 4, 2021 15:56

xddxdd changed the title ~~Support fetch external resources over HTTP~~ [WIP] Support fetch external resources over HTTP Jan 4, 2021

add fetch and various extensions to otto

349cd50

add PANIC() and error-handled FETCH()

xddxdd force-pushed the pr-fetch branch from 1ba9808 to 349cd50 Compare January 5, 2021 17:30

xddxdd changed the title ~~[WIP] Support fetch external resources over HTTP~~ Support fetch external resources over HTTP Jan 5, 2021

tlimoncelli approved these changes Jan 5, 2021

View reviewed changes

Merge branch 'master' into pr-fetch

1695c7f

tlimoncelli merged commit 6efedd6 into DNSControl:master Jan 6, 2021

xddxdd deleted the pr-fetch branch November 14, 2022 07:32

Conversation

xddxdd commented Dec 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xddxdd commented Dec 26, 2020

Uh oh!

tlimoncelli left a comment

Choose a reason for hiding this comment

Uh oh!

tlimoncelli Dec 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xddxdd commented Jan 1, 2021

Uh oh!

tlimoncelli commented Jan 4, 2021

Uh oh!

xddxdd commented Jan 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

xddxdd commented Jan 4, 2021

Uh oh!

tlimoncelli commented Jan 4, 2021

Uh oh!

xddxdd commented Jan 5, 2021

Uh oh!

tlimoncelli left a comment

Choose a reason for hiding this comment

Uh oh!

xddxdd commented Jan 6, 2021

Uh oh!

tlimoncelli commented Jan 6, 2021

Uh oh!

patschi commented Jan 8, 2021

Uh oh!

xddxdd commented Jan 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

xddxdd commented Dec 26, 2020 •

edited

Loading

tlimoncelli Dec 28, 2020 •

edited

Loading

xddxdd commented Jan 4, 2021 •

edited

Loading