New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support protocols implemented in other languages #174

Open
bbyars opened this Issue Dec 4, 2016 · 4 comments

Comments

1 participant
@bbyars
Owner

bbyars commented Dec 4, 2016

JDBC is an example that would be difficult to implement purely in node, but should be quite reasonable implementing in Java. A protocol that shelled out, taking the imposter config as stdin and output the response, could make for interesting extension.

It feels like custom protocols should be passed on the command line, since they're a security vulnerability when shelled out through the API if it's called remotely:

mb --proto jdbc jdbc.jar

Then you could call it as normal:

{
  "protocol": "jdbc",
  "stubs": [...]
}

mountebank could ship with core ones like jdbc that hide the fact that they shell out from the user, while still allowing new ones to be tested before inclusion in the core.

@bbyars

This comment has been minimized.

Owner

bbyars commented Dec 4, 2016

Worth thinking about how to shell out request separately from response. If we could shell out a TCP packet, for example, to get the mountebank request format, then we could use the existing predicate behavior, as well as matches array, requests array, etc. The trick would be defining the end of the response, so it might need an endOfResponseResolver in javascript.

A second shell call could generate the response.

@bbyars bbyars added this to the v1.9.0 milestone Jan 5, 2017

@bbyars bbyars changed the title from Support protocol that shells out the implementation to Support protocols implemented in other languages Jan 7, 2017

@bbyars

This comment has been minimized.

Owner

bbyars commented Jan 7, 2017

After further thought, handing off TCP packets to another process and expecting them to transform that into a mountebank request means not taking advantage of server libraries written in other languages. If I had written http by handing off TCP packets to another process, I wouldn't have been able to use http server code; I would have had to use TCP code and completely re-implement an http server.

An alternative architecture may be needed. For all protocols, mountebank could fork a new process that spins up an actual (http, tcp, ldap, jdbc, etc) server. The core mountebank server (port 2525) could add an endpoint that translates a mountebank-encoded request (e.g. just the fields the user cares about) to a mountebank-encoded response, which the actual server could transform into a real network response. That means all behaviors and predicates would be handled by mountebank, as well as managing the state of requests for the --mock flag, and all network management could be handled by the other process. Logging could either use the stdout file handle given by mountebank, or use a mountebank endpoint from the protocol server to mountebank. mountebank could stop the process to kill it, meaning there would not need to be any admin endpoints on the protocol itself (I think).

@bbyars

This comment has been minimized.

Owner

bbyars commented Jan 8, 2017

Protocols

Originally, mountebank created imposters in-process, simply by opening up a new socket of the given
protocol. This was an elegant solution so long as the servers could be easily written in node. There
are a number of protocols that are simply not feasible to implement in node, but easy to implement
in other languages (like JDBC which is Java based, or SOAP which has poor node support).

Supporting more protocols means making protocols easier to add. This requires removing any hooks
from the mountebank codebase and creating a standard interface that new protocols could implement
in different codebases. To allow those codebases language flexibility, they would have to be
different processes. This document is my attempt at capturing the places that need to change
to make this architecture possible.

Imposter Management

mountebank.js initializes the protocol implementations with allowInjection, mock, and debug.
That returns

{
    name: 'foo',
    create: function (config) { return createServer(config, mock, debug); },
    Validator: {
      create: function () { return DryRunValidator.create(); }
    }
}

The next interaction is the request to http://localhost:2525/imposters, which validates the request
using the Validator. Validation may need to remain in mountebank unless there's a clear
separation possible.

The call to create occurs in imposter.js. That should be changed to shell out. The only
information needed for the protocol would be root level information in the imposter JSON. All
responses, predicates, and behaviors can be managed by mountebank in a protocol-agnostic
fashion. Root level information includes the port, name, and any protocol-specific metadata
like https certificates and tcp mode. These should be passed as key-value pairs on the
command line (e.g. httpProtocol --port 3000). We may need different Windows vs Unix
command lines (e.g. httpProtocol vs node httpProtocol).

The protocol should accept connections and simplify each request to the protocol-specific
request format that is documented on the protocol page in mountebank. The JSON may be
quite large, so it should then POST the payload to http://localhost:2525/imposters/3000/requests.
mountebank should use the predicates to pick the correct response, apply behaviors to it,
and send the response JSON in the HTTP response to the POST. The protocol process, which
could have multiple sockets open, should translate the simplified response into a real network
response.

Proxying adds complexity. For a proxy response, mountebank should respond to the call to
http://localhost:2525/imposters/3000/requests with the proxy configuration and a link to
the index in the requests array (or numberOfRequests), something like
http://localhost:2525/imposters/3000/requests/15, without applying behaviors. After the
protocol receives the proxy response, it should POST the JSON response to that address,
at which point mountebank applies the behaviors and sends back the transformed response.
This means that there need no special http endpoint; all communication is from the protocol
implementation to mountebank.

Design Constraints

  • All communication will be from the protocol implementation to mountebank. As discussed below,
    this means that there is no obvious solution for full migration of protocol logic to
    the protocol implementation (like documentation), but it simplifies the architecture
    considerably. The alternative would be to force non-HTTP protocol implementations to
    expose an HTTP endpoint, and for HTTP implementations to either have a magic endpoint which
    risks interference with virtualized endpoints or create another socket, which risks
    collision with a needed port for virtual services.
  • mountebank remains in control of predicates, response resolution, and behaviors, and
    remains completely ignorant of any network translation of those responses
  • There is only one mountebank log. Users do not have to view separate logs for separate
    imposters
  • Performance is a concern, especially for large payloads. Keeping everything on the
    loopback interface should help. There may need to be performance tweaks, like not requiring
    the two-step dance on proxies if there are no behaviors, and minimizing the number
    of log entry calls by examining the log level

Shell Implications

I'm concerned that the process creation could create a race condition. As it stands right now,
the imposter is usable as soon as the promise resolves in mountebank. With process creation,
it may not be bound to the socket when the process creation method returns.

Something the mountebank process itself does is to rely on the pidfile creation to remove
race conditions for build processes. I don't know if that makes sense -- if the protocol
implementation should write a pidfile -- but it's one option.

It's also unclear how mountebank will handle the DELETE request for an option. Should it
kill the process in the same way as mb, with the kill operating system command using
a pidfile, or kill the process as a child process using node's API?

Extensions

There are four interaction points not well covered by the architecture above: documentation,
validation, logging, and a UI to create imposters (on /imposters). I can't think of a way to migrate
those to the protocol implementation except by adding HTTP endpoints exposing them, which I
don't want to do. At least for this first pass, each of these concerns will stay in the
core mountebank codebase.

Logging may be handled by capturing the stdout stream of the child process

@bbyars

This comment has been minimized.

Owner

bbyars commented Jan 10, 2017

Some of the concerns above could be solved by having each protocol exist as a library and an optional application. The node library could be responsible for starting and stopping (which maybe could even be in-process), ensuring that the app is bound to the socket before returning (e.g by having the app write to stdout), validation, documentation, etc. The app could work as described above.

@bbyars bbyars removed this from TODO in v1.9.0 Jan 31, 2017

@bbyars bbyars added this to TODO in v1.10.0 Jan 31, 2017

@bbyars bbyars modified the milestones: v1.10.0, v1.9.0 Feb 9, 2017

@bbyars bbyars removed this from TODO in v1.10.0 Feb 27, 2017

@bbyars bbyars modified the milestones: v1.11.0, v1.10.0 Feb 27, 2017

@bbyars bbyars added this to Backlog in v1.11.0 Feb 27, 2017

@bbyars bbyars removed this from Backlog in v1.11.0 May 19, 2017

@bbyars bbyars modified the milestones: v1.13.0, v1.11.0 May 19, 2017

@bbyars bbyars added this to v1.13.0 in Roadmap May 20, 2017

@bbyars bbyars moved this from v1.13.0 to v1.15.0 in Roadmap Jul 22, 2017

@bbyars bbyars removed this from the v1.13.0 milestone Feb 18, 2018

@bbyars bbyars moved this from v1.15.0 to v1.16.0 in Roadmap Feb 18, 2018

@bbyars bbyars added this to Wider protocol support in Themes Feb 18, 2018

@bbyars bbyars moved this from v1.16.0 to v1.17.0 in Roadmap Feb 18, 2018

@bbyars bbyars moved this from v1.17.0 to v1.16.0 in Roadmap Feb 18, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment