The big refactor! #200

daviddias · 2018-02-07T11:54:45Z

The current ipfsd-ctl is a big mess. There are no only a few browser tests; tests have the setup assert and teardown phases all mixed with sometimes using a describe + before as the assertion; there are multiple files that get loaded in the browser without necessity, this is causing all sorts of problems on other modules ipfs-inactive/js-ipfs-http-client#683; The internals use different names for the things that are exposed (i.e calling daemon-ctrl to the Daemon Factory).

There is still a considerable amount of work to do here. I might not be able to finish this today but I intend to have it tomorrow.

dryajov · 2018-02-07T16:11:25Z

@diasdavid I'm not sure what you mean by:

There are no browser tests;

All tests run in the browser, what other browser tests are missing?

daviddias · 2018-02-07T16:53:19Z

I take that back, there is indeed a suite that gets run in the browser https://github.com/ipfs/js-ipfsd-ctl/blob/master/test/browser.js#L4

dryajov · 2018-02-07T17:47:35Z

@diasdavid right, the rest of the tests are concerned with daemon specific tests (i.e. repo exist in the filesystem, daemon cleans up correctly when killed, executable is able to resolve correctly, etc... which can't be tested from the browser).

Thanks for the refactor 👍 I think it cleans up a lot of things.

victorb · 2018-02-09T09:31:20Z

.aegir.js

@@ -2,7 +2,8 @@

 const createServer = require('./src').createServer

-const server = createServer()
+const server = createServer({ port: 25000 })


Can we use port 0 to avoid port collisions?

This is spawned in a separate process by aegir, we would have to create yet another side channel just to pass this port over. Not worth it. For other cases, yes for port 0.

daviddias

Made good progress this morning, starting to see the light at the end of the tunnel :)

The commit history is a mess because I had to change so many things that at some point it became meaningless.

Still some work TODO though. The project is finally in a state where the structure feels right and it is simple to isolate tests and bugs. I won't be able to touch this again until tomorrow morning. If you have time to contribute, please do :)

daviddias · 2018-02-09T12:26:39Z

README.md

-    | +-----------------------+   |                    +-----------------------|----                      |  |
-    |                             |                                            |   +----------------------+  |
-    +-----------------------------+                                            +-----------------------------+
-```


Needs a new diagram.

What is the current diagram missing?

daviddias · 2018-02-09T12:27:06Z

README.md

+    └── tmp-dir.js
+
+4 directories, 21 files
+```


I'm finally happy with the code structure :)

Here for reference:

src ├── defaults │ ├── config.json │ └── options.json ├── endpoint # endpoint to support remote spawning │ ├── routes.js │ └── server.js ├── factory-client.js # IPFS Factories: client (remote), daemon (go or js) and in-proc (js) ├── factory-daemon.js ├── factory-in-proc.js ├── index.js ├── ipfsd-client.js # ipfsd (Daemon Controller): client (remote), daemon (go or js), in-proc (js) ├── ipfsd-daemon.js ├── ipfsd-in-proc.js └── utils # Utils used by the Factories and Daemon Controllers ├── configure-node.js ├── exec.js ├── find-ipfs-executable.js ├── flatten.js ├── parse-config.js ├── repo │ ├── create-browser.js │ └── create-nodejs.js ├── run.js ├── set-config-value.js └── tmp-dir.js 4 directories, 21 files

daviddias · 2018-02-09T12:27:39Z

src/endpoint/routes.js

-      const ctrl = new CtrlFactory({ type: payload.type })
-      ctrl.spawn(payload.opts, (err, ipfsd) => {
+
+      // TODO: use the ../src/index.js so that the right Factory is picked


daviddias · 2018-02-09T12:27:50Z

src/factory-daemon.js

+    }
+    options = Object.assign({}, options, { type: this.type })
+    // TODO: (1) this should check to see if it is looking for Go or JS
+    // TODO: (2) This spawns a whole daemon just to get his version? There is


just doing new Daemon won't actually start the daemon, it will simply allow you to send commands to the selected executable.

It still should check if it is JS or Go

Why? All the logic around daemon type is already being handled in the Daemon itself - not sure what to do with the check here.

daviddias · 2018-02-09T12:28:03Z

src/factory-in-proc.js

+      callback = options
+      options = {}
+    }
+    // TODO create this method


daviddias · 2018-02-09T12:28:29Z

src/ipfsd-daemon.js

@@ -323,7 +327,8 @@ class Node {
      ),
      (config, cb) => {
        if (!key) {
-          return tryJsonParse(config, cb)
+          // TODO: this function didn't exist on the original codebase
+          // return tryJsonParse(config, cb)


Figure out why this was here in the first place.

because the daemon process will return a json string back, not an object.

@dryajov even if that is correct, the tryJsonParse function doesn't seem to exist anywhere in the codebase

daviddias · 2018-02-09T12:28:40Z

test/start-stop.node.js

+        expect(ipfsd).to.exist()
+      })
+
+      // TODO fix this test


daviddias · 2018-02-09T12:28:57Z

test/start-stop.node.js

+      it('.start with flags', (done) => {
+        // TODO js-ipfs doesn't fail on unrecognized args. Think what should be
+        // the desired behaviour
+        if (type === 'js') { return this.skip() }


dryajov · 2018-02-09T16:14:42Z

src/factory-in-proc.js

@@ -3,44 +3,20 @@
 const defaults = require('lodash.defaultsdeep')


why duplicate if factory-in-proc share %90+ code with factory-daemon?

We should have a base class then. The main reason is that you want to avoid having factory-daemon being loaded in the browser (and all of its dependencies).

dryajov · 2018-02-09T16:22:15Z

test/add-retrieve.spec.js

@@ -0,0 +1,63 @@
+/* eslint-env mocha */


This was originally intended as a set of reusable tests to run after the different combinations of spawn/init/start to make sure the daemon were being created in a functional way - this intent is lost with this change. How are we testing for this cases?

dryajov · 2018-02-09T16:31:49Z

test/spawn-options.node.js

+      df = DaemonFactory.create(dfOpts)
+    })
+
+    // TODO: why does ipfsd-ctl polute the env variables?


It used to set IPFS_PATH and other variables, its not the case anymore - we can remove that.

dryajov · 2018-02-09T16:32:37Z

test/spawn-options.node.js

+        const options = {
+          init: false,
+          start: false,
+          disposable: true


this needs to be false

dryajov · 2018-02-09T16:35:56Z

test/spawn-options.node.js

+    })
+
+    // TODO ?? What is this test trying to prove?
+    describe('spawn a node and attach api', () => {


confusing name for the test - its trying to make sure that starting the node withspawn with the default options will return a fully constructed ipfsd object. The reusable tests I pointed to above where there to make sure it was also started and initialized correct and permitted to interact with it over the attached ipfs-api.

dryajov · 2018-02-09T16:38:14Z

test/spawn-options.node.js

+          }
+        ], (err) => {
+          expect(err).to.not.exist()
+          expect(fs.existsSync(repoPath)).to.be.ok()


// TODO why wouldn't this work from the browser if we use the
// remote endpoint?

because we use fs.existsSync - is there a way of making it work in the browser?

dryajov · 2018-02-09T22:58:49Z

@diasdavid I took on fixing some of the outstanding todos as well as making sure all tests pass in node.

daviddias · 2018-02-10T07:59:37Z

README.md


 ## Table of Contents

 - [Install](#install)
 - [Usage](#usage)
 - [API](#api)
+- [Packageing](#packaging)


daviddias · 2018-02-10T08:01:14Z

src/factory-client.js

+      callback = options
+      options = {}
+    }
+    // TODO implement this method


This method needs to be implemented as well for clients. Once it is, enable tests again.

In progress, will push shortly.

daviddias · 2018-02-10T08:02:02Z

src/factory-daemon.js

+    }
+    options = Object.assign({}, options, { type: this.type })
+    // TODO: (1) this should check to see if it is looking for Go or JS
+    // TODO: (2) This spawns a whole daemon just to get his version? There is


It still should check if it is JS or Go

daviddias · 2018-02-12T08:13:36Z

Ok, fixed some of the tests, now I'm seeing a new issue. It seems that when spawning a js-ipfs daemon and/or in-proc node, it doesn't respect the default config of ipfsd-ctl and uses the default of js-ipfs causing port clashes

daviddias · 2018-02-12T08:24:04Z

src/factory-in-proc.js

      const nonDisposableConfig = clone(defaultConfig)
-      delete nonDisposableConfig.Addresses
+      // TODO why delete the addrs here???
+      // delete nonDisposableConfig.Addresses


@dryajov why were you deleting the addrs here?

Because at some point we wanted non-disposable nodes to use js-ipfs or go-ipfs default ports and not random ports.

daviddias · 2018-02-12T08:38:31Z

src/factory-daemon.js

@@ -26,7 +26,7 @@ class FactoryDaemon {
    if (options && options.type === 'proc') {
      throw new Error('This Factory does not know how to spawn in proc nodes')
    }
-    options = Object.assign({ type: 'go' }, options)
+    options = Object.assign({}, { type: 'go' }, options)


@dryajov why this change?

daviddias · 2018-02-12T08:43:05Z

src/factory-in-proc.js

@@ -4,11 +4,14 @@ const defaults = require('lodash.defaultsdeep')
 const clone = require('lodash.clone')
 const waterfall = require('async/waterfall')
 const path = require('path')
+const tmpDir = require('./utils/tmp-dir')


This brought the os module to the browser.

daviddias · 2018-02-12T12:01:17Z

Woot! Good to merge. Deferring remaining tune up to later PRs.

Also tested with js-ipfs-api and ipfs/interop

part I of the big refactor!

012c327

ghost assigned daviddias Feb 7, 2018

ghost added the status/in-progress In progress label Feb 7, 2018

foundation for browser tsts

5040d41

SLAAAAY the 🐉🐉🐉

df9ffa0

victorb reviewed Feb 9, 2018

View reviewed changes

daviddias added 7 commits February 9, 2018 09:44

add-retrieve

caca0e2

test: api.node

d72ab8a

docs: trim the readme

06f70a2

moar refactor

2582dbf

test: remote endpoint tests

9f6632c

start spinning browser tests again

56c618e

I am finally happy with the structure of this project

b0da61a

daviddias commented Feb 9, 2018

View reviewed changes

dryajov reviewed Feb 9, 2018

View reviewed changes

test/spawn-options.node.js Outdated

const options = {

init: false,

start: false,

disposable: true

Copy link

Member

dryajov Feb 9, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this needs to be false

dryajov reviewed Feb 9, 2018

View reviewed changes

fix tests in node as well as resolve a couple of todos

a9f6031

ghost assigned dryajov Feb 9, 2018

dryajov added 3 commits February 9, 2018 16:23

feat: implemented the version method for factory-in-proc

7c149a1

fix: daemon exec path should match type test

664d8ed

lint

e476440

doc

aab54af

daviddias commented Feb 10, 2018

View reviewed changes

daviddias mentioned this pull request Feb 10, 2018

Fix 683 ipfs-inactive/js-ipfs-http-client#688

Merged

dryajov and others added 5 commits February 10, 2018 10:48

feat: added version endpoint

2a00842

experiment

37194d3

feat: add tmpDir endpoint to be able to generate tmp repos in browser

7ee051f

chore

97ab8ed

test: fix spawn options tests

581945e

daviddias force-pushed the ipfsd-light branch from eb51dab to 581945e Compare February 12, 2018 08:05

dont delete addrs

4d87386

daviddias commented Feb 12, 2018

View reviewed changes

daviddias added 2 commits February 12, 2018 08:29

docs: polish the readme

8cc19eb

chore

12ef363

daviddias commented Feb 12, 2018

View reviewed changes

daviddias added 11 commits February 12, 2018 08:50

test: all running

63f5243

test: all passing locally

5738822

bump timeouts

bea0caf

test: add prefix back to repo name

1463b98

chore

19a4133

chore: timeout dance

ce7b68c

fix: tmp dir

d2f8680

real timeout

d9ada2e

feat: do not depend on any locally initalized daemon

dd20ef7

chore

30c25c6

now yes

f9d3eb4

daviddias merged commit adbef1b into master Feb 12, 2018

daviddias deleted the ipfsd-light branch February 12, 2018 12:01

ghost removed the status/in-progress In progress label Feb 12, 2018

		@@ -3,44 +3,20 @@
		const defaults = require('lodash.defaultsdeep')

The big refactor! #200

The big refactor! #200

Conversation

daviddias commented Feb 7, 2018 • edited Loading

dryajov commented Feb 7, 2018

daviddias commented Feb 7, 2018

dryajov commented Feb 7, 2018

Choose a reason for hiding this comment

daviddias Feb 9, 2018 • edited Loading

Choose a reason for hiding this comment

daviddias left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dryajov Feb 10, 2018 • edited Loading

Choose a reason for hiding this comment

daviddias Feb 9, 2018 • edited by dryajov Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dryajov commented Feb 9, 2018

daviddias Feb 10, 2018 • edited by dryajov Loading

Choose a reason for hiding this comment

daviddias Feb 10, 2018 • edited by dryajov Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias commented Feb 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias commented Feb 12, 2018 • edited Loading

daviddias commented Feb 7, 2018 •

edited

Loading

daviddias Feb 9, 2018 •

edited

Loading

daviddias left a comment •

edited

Loading

dryajov Feb 10, 2018 •

edited

Loading

daviddias Feb 9, 2018 •

edited by dryajov

Loading

daviddias Feb 10, 2018 •

edited by dryajov

Loading

daviddias Feb 10, 2018 •

edited by dryajov

Loading

daviddias commented Feb 12, 2018 •

edited

Loading