Distributed Press Monetization API responese on DWeb #25

benhylau · 2020-12-19T03:47:59Z

🎟️ Re-ticketed from: #
📅 Due date: 21 Dec 2020
🎯 Success criteria: Get Monetizational API to respond to IPFS and Hypercore as according to API spec #23.

Task Summary

Periodically pin https://api.distributed.press/compost/v0/monetization/balances responses onto the IPFS and Hypercore networks. Make sure 15-min up-to-date responses are available over ipfs://, ipns://, and hyper://.

These periodic pinning belong to Pinning Services in Distributed Press repo: https://github.com/hyphacoop/api.distributed.press#architecture

To Do

The text was updated successfully, but these errors were encountered:

benhylau · 2020-12-20T20:04:40Z

IPFS

See ipfs/in-web-browsers#174 for Opera Android resolution issue for IPFS DNSLink, but otherwise this works now on supported browsers:

ipns://api.staging.compost.digital/v0/monetization/balances.json

Also note ipfs/in-web-browsers#169 when trying to access via HTTP using subdomain-based gateways. This seems like a priority that IPFS will address early next year.

Hypercore

We will need DEP-0005: DNS support in Beaker browser. Right now this only works in Agregore:

hyper://api.staging.compost.digital/v0/monetization/balances.json

If someone wants to test out these links, your best bet is probably to download Agregore as it has support for everything here.

benhylau · 2020-12-21T03:15:57Z

@RangerMauve I wrote the following Promise:

function publishHyperdrive(seed, dir) {
  return new Promise((resolve, reject) => {
    // Publish hyperdrive to dat-store
    hyperPublisher.getURL({seed})
      .then(url => datStoreClient.add(url)) <-- PAY ATTENTION TO THIS LINE *
      .then(hyperPublisher.create({seed}))
      .then(hyperPublisher.sync({
        seed,
        fsPath: dir,
        drivePath: '/'
      }))
      .then((diff, url) => {
        console.log('diff');
        console.log(diff);
        console.log('url');
        console.log(url);
        resolve(url);
      });
  });
}

If I run this as is, I get this:

diff
{}
url
undefined

If I comment out line *, I get this:

diff
{
  url: 'hyper://4058f494eacf0d54ad11536bf5a31ab73da6ed209c55a5db8530a7534eb9d8fa'
}
url
undefined

In either case, my hyperdrive isn't syncing with the file system. Any idea?

I also have these in the dat-store:

$ dat-store list
hyper://4058f494eacf0d54ad11536bf5a31ab73da6ed209c55a5db8530a7534eb9d8fa - hyper://4058f494eacf0d54ad11536bf5a31ab73da6ed209c55a5db8530a7534eb9d8fa
hyper://4058f494eacf0d54ad11536bf5a31ab73da6ed209c55a5db8530a7534eb9d8fa - /home/compost/api.distributed.press/pinning-services/stuff

The first entry was created by hyper-publisher, and the latter manually via dat-store CLI. Neither one is accessible in the local proxy or in Beaker/Agregore, so I suspect nothing is syncing.

This local http proxy should work right?

https://hyper.distributed.press/4058f494eacf0d54ad11536bf5a31ab73da6ed209c55a5db8530a7534eb9d8fa

RangerMauve · 2020-12-21T03:28:00Z

Try setting verbose:true to get some console output. Are you sure the dir is being passed in properly? It looks like there's an empty folder there or something

benhylau · 2020-12-21T03:33:50Z

Nothing suspicious turning on verbose:

Connected {
  remoteAddress: '198.50.215.6',
  remoteType: 'tcp',
  remotePublicKey: <Buffer f3 a2 c8 e5 73 4a 7c 0e 66 5b cf b6 f4 74 32 0a 6f 77 5f b9 22 80 a9 fc a0 01 f4 12 4c 4d 77 65>
}
Waiting to sync metadata
Peer has header true
Connected {
  remoteAddress: '198.50.215.6',
  remoteType: 'tcp',
  remotePublicKey: <Buffer f3 a2 c8 e5 73 4a 7c 0e 66 5b cf b6 f4 74 32 0a 6f 77 5f b9 22 80 a9 fc a0 01 f4 12 4c 4d 77 65>
}
Waiting for update 14
Initializing Hyperdrive
Finding changed files
No new changes detected { diff: [] }
Done
Seed:
[REDACTED]
URL:
hyper://4058f494eacf0d54ad11536bf5a31ab73da6ed209c55a5db8530a7534eb9d8fa
Starting sync
hyper://4058f494eacf0d54ad11536bf5a31ab73da6ed209c55a5db8530a7534eb9d8fa
Listening for peers
diff
{
  url: 'hyper://4058f494eacf0d54ad11536bf5a31ab73da6ed209c55a5db8530a7534eb9d8fa'
}
url
undefined

benhylau · 2020-12-21T04:48:48Z

@RangerMauve thanks for hopping onto the call to get everything sorted! Here's what we gotta do:

@ASoTNetworks get rid of the gateway in https://hyper.distributed.press/gateway/4058f494eacf0d54ad11536bf5a31ab73da6ed209c55a5db8530a7534eb9d8fa/v0/monetization/balances.json on Nginx
@ASoTNetworks open up 42666 (see link)

preferredPort: 42666 **Node**: The port hyperswarm should try to bind on. You should allow it through your firewall on TCP/UDP for best results.
@benhylau to stop using Promise like an idiot :) I was running the calls in parallel and returning garbage
@benhylau to only do this on first run, otherwise create locks up
- hyperPublisher.getURL
- datStoreClient.add
- hyperPublisher.create
@benhylau place a mutex lock on hyperPublisher.sync otherwise if two processes sync on same seed the history may fork

benhylau · 2020-12-21T05:56:11Z

@RangerMauve it turns out dat-store remove <url|path> hangs the service, regardless of whether I pass a url or path, and I have to kill the process otherwise all the commands to dat-store would lock up.

RangerMauve · 2020-12-21T06:22:07Z

You might want to go to the storage location of dat-store on the VM and manually remove the url from URLs.json and folders.json

benhylau · 2020-12-21T07:55:40Z

Yup the manual removals worked. I successfully published everything once, but now I keep getting this, even after restarting the dat-store process:

Starting sync
hyper://1ff986e2999f9670d17121bef3fa922ac2da52b17061b8f75ea36a7ec556cf00
Listening for peers
Peer has header false
Connected {
  remoteAddress: '198.50.215.6',
  remoteType: 'tcp',
  remotePublicKey: <Buffer f3 a2 c8 e5 73 4a 7c 0e 66 5b cf b6 f4 74 32 0a 6f 77 5f b9 22 80 a9 fc a0 01 f4 12 4c 4d 77 65>
}
Waiting for update 0
Error: Unable to get Update
    at Object.sync (/home/compost/api.distributed.press/pinning-services/node_modules/hyperdrive-publisher/index.js:141:13)
    at processTicksAndRejections (node:internal/process/task_queues:93:5)

Something failing in the sync -> https://github.com/RangerMauve/hyperdrive-publisher/blob/default/index.js#L141

Unrelated to this, is it necessary to call sync at all? Wouldn't dat-store automatically observe changes?

Edit: Now everything works again, not sure what changed.

benhylau · 2020-12-21T08:30:08Z

@RangerMauve can you verify the following work on Agregore?

hyper://e207dff7087f254174b51609e4ab552dc08b9ef28d36f5885fac34d0671358b1/v0/monetization/balances.json
hyper://api.staging.compost.digital/v0/monetization/balances.json (not supported on Beaker)

After gateway fix:

https://hyper.distributed.press/e207dff7087f254174b51609e4ab552dc08b9ef28d36f5885fac34d0671358b1/v0/monetization/balances.json
(with patch Add DNS TXT record resolution in gateway RangerMauve/dat-store#48) https://hyper.distributed.press/api.staging.compost.digital/v0/monetization/balances.json
- it seems dat-store gateway doesn't support domains via TXT records
- @ASoTNetworks is it possible to turn api.staging.compost.digital -> e207dff7087f254174b51609e4ab552dc08b9ef28d36f5885fac34d0671358b1 in Nginx with a TXT lookup?

RangerMauve · 2020-12-21T19:19:45Z

@benhylau The URLs you linked to work in Agregore.

I'm not sure if dat-store had DNS support for resolving URLs, but that might be something you could find in the library.js file. I'd be down to review a PR adding it in if it's missing, but I'm kinda short on gratis time this week since I have contracts to work on. 😅 Could probably spare a couple of hours of consulting time though.

benhylau · 2020-12-22T07:22:37Z

On Agregore, DNS doesn't work for me but it seems to work for you, so maybe a MacOS thing?

About dat-store support for TXT, it seems to not be supported in the gateway. I did this and it seems to work RangerMauve/dat-store#48.

RangerMauve · 2020-12-22T21:16:30Z

The dat-store fix looks good.

I'm honestly not sure why the DNS stuff wouldn't be working. :/ What did you set the TXT record to exactly?

Does hyper://blog.mauve.moe work?

Might make sense to clone dat-dns and put your domain in there to see if the formatting is valid

RangerMauve · 2020-12-22T21:17:57Z

Here's the info about the DNS txt record for reference: https://github.com/datprotocol/DEPs/blob/master/proposals/0005-dns.md#dns-txt-record

benhylau · 2020-12-22T21:49:24Z

Does hyper://blog.mauve.moe work?

Apparently yes! lol

What did you set the TXT record to exactly?

dig api.staging.compost.digital TXT +short
"datkey=e207dff7087f254174b51609e4ab552dc08b9ef28d36f5885fac34d0671358b1"

blog.mauve.moe is using a different method in DEP-0005: DNS:

$ dig blog.mauve.moe TXT +short
$ dig blog.mauve.moe A +short
165.227.36.113
$ curl http://blog.mauve.moe/.well-known/dat
dat://94f0cab7f60fcc2a711df11c85db5e0594d11e8a3efd04a06f46a3c34d03c418
ttl=3600

It appears that Agregore supports .well-known/dat, but not the DNS TXT record.

benhylau · 2020-12-28T08:08:07Z

Both of these work in Agregore now since we fixed the dat-store:

hyper://staging.compost.digital
hyper://api.staging.compost.digital/v0/monetization/balances.json

benhylau · 2021-01-08T06:23:56Z

Response updated to include total estimated balance.

benhylau added the COMPOST: Technical Magazine software and website development label Dec 19, 2020

benhylau self-assigned this Dec 19, 2020

benhylau mentioned this issue Dec 22, 2020

Add DNS TXT record resolution in gateway RangerMauve/dat-store#48

Merged

benhylau mentioned this issue Dec 22, 2020

Domain resolution via DNS TXT records AgregoreWeb/agregore-browser#90

Closed

benhylau closed this as completed Jan 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Distributed Press Monetization API responese on DWeb #25

Distributed Press Monetization API responese on DWeb #25

benhylau commented Dec 19, 2020 •

edited

benhylau commented Dec 20, 2020 •

edited

benhylau commented Dec 21, 2020 •

edited

RangerMauve commented Dec 21, 2020

benhylau commented Dec 21, 2020

benhylau commented Dec 21, 2020 •

edited by ASoTNetworks

benhylau commented Dec 21, 2020

RangerMauve commented Dec 21, 2020

benhylau commented Dec 21, 2020 •

edited

benhylau commented Dec 21, 2020 •

edited

RangerMauve commented Dec 21, 2020 •

edited

benhylau commented Dec 22, 2020

RangerMauve commented Dec 22, 2020

RangerMauve commented Dec 22, 2020

benhylau commented Dec 22, 2020

benhylau commented Dec 28, 2020

benhylau commented Jan 8, 2021

Distributed Press Monetization API responese on DWeb #25

Distributed Press Monetization API responese on DWeb #25

Comments

benhylau commented Dec 19, 2020 • edited

Task Summary

To Do

benhylau commented Dec 20, 2020 • edited

IPFS

Hypercore

benhylau commented Dec 21, 2020 • edited

RangerMauve commented Dec 21, 2020

benhylau commented Dec 21, 2020

benhylau commented Dec 21, 2020 • edited by ASoTNetworks

benhylau commented Dec 21, 2020

RangerMauve commented Dec 21, 2020

benhylau commented Dec 21, 2020 • edited

benhylau commented Dec 21, 2020 • edited

RangerMauve commented Dec 21, 2020 • edited

benhylau commented Dec 22, 2020

RangerMauve commented Dec 22, 2020

RangerMauve commented Dec 22, 2020

benhylau commented Dec 22, 2020

benhylau commented Dec 28, 2020

benhylau commented Jan 8, 2021

benhylau commented Dec 19, 2020 •

edited

benhylau commented Dec 20, 2020 •

edited

benhylau commented Dec 21, 2020 •

edited

benhylau commented Dec 21, 2020 •

edited by ASoTNetworks

benhylau commented Dec 21, 2020 •

edited

benhylau commented Dec 21, 2020 •

edited

RangerMauve commented Dec 21, 2020 •

edited