Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create countertop topology generation #83

Merged
merged 5 commits into from Aug 17, 2020
Merged

Conversation

slifty
Copy link
Member

@slifty slifty commented Aug 5, 2020

Description

This PR adds the first portion of the countertop, which is the central workspace of the TV kitchen ultimately responsible for setting up the appliances and data streams to process all unique data streams.

Specifically, it adds the countertop components necessary to register appliances and convert a set of registered appliances into a data flow topology (the various data paths between sections of the countertop which will transform that data into new formats).

This begins to implement some of the API described in #82.

Since this is the first countertop PR, it defines a few key classes which will be expanded on in future PRs:

  • CountertopCoordinator defines the developer API -- developers will be working with this entity directly. It is where new appliances are registered. It generates the rest of the countertop.

  • CountertopStation represents a section of the countertop dedicated to a single configured appliance. There will likely be more than one copy of a given appliance (one copy per data stream).

  • CountertopStream is an object that is used to (A) determine how many appliances should exist in a given station and (B) determine which appliances should process a given payload.

Issue #23 goes through a fair number of example topologies.

This PR does not completely finish the countertop, but rather it creates the API for registering appliances and converts a list of appliances into a topology which the countertop will use to set up stations and route packages between stations.

Due Diligence Checklist

  • Tests
  • Documentation
  • Consolidated commits
  • Relevant labels assigned

Steps to Test

The best way to test this is using your _sandbox.js

Here's an example sandbox:


import { IAppliance } from '@tvkitchen/base-interfaces'
import CountertopCoordinator from '%src/components/countertop/CountertopCoordinator'

class SourceAppliance extends IAppliance {
	static getInputTypes = () => []
	static getOutputTypes = () => ["STREAM.CONTAINER"]
}

class CaptionAppliance extends IAppliance {
	static getInputTypes = () => ["STREAM.CONTAINER"]
	static getOutputTypes = () => ["TEXT.ATOM"]
}

class LineAppliance extends IAppliance {
	static getInputTypes = () => ["TEXT.ATOM"]
	static getOutputTypes = () => ["TEXT.LINE"]
}

class ComplexAppliance extends IAppliance {
	static getInputTypes = () => ["TEXT.LINE", "STREAM.CONTAINER"]
	static getOutputTypes = () => ["IMAGE.FILE"]
}

const countertop = new CountertopCoordinator()
countertop.addAppliance(SourceAppliance)
countertop.addAppliance(SourceAppliance)
countertop.addAppliance(CaptionAppliance)
countertop.addAppliance(LineAppliance)
countertop.addAppliance(ComplexAppliance)
const topology = countertop.updateTopology()
console.log(topology)

Deploy Notes

None

Related Issues

@slifty slifty force-pushed the 30-create-stubbed-sous-chef branch 2 times, most recently from d6e9f2a to 2786c51 Compare August 7, 2020 18:53
@slifty slifty requested a review from chriszs August 7, 2020 18:58
@slifty
Copy link
Member Author

slifty commented Aug 7, 2020

@chriszs This is not final, but what's in here is ready for review. Specifically the toplogy generation.

I'm going to be adding tests and documentation, but 85% of the complexity of the PR is already in here.

src/components/countertop/CountertopCoordinator.js Outdated Show resolved Hide resolved
src/components/countertop/CountertopCoordinator.js Outdated Show resolved Hide resolved
src/components/countertop/CountertopStation.js Outdated Show resolved Hide resolved
src/components/countertop/CountertopStation.js Outdated Show resolved Hide resolved
Copy link
Contributor

@chriszs chriszs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still reviewing, but here's what I've got so far.

src/components/countertop/CountertopStation.js Outdated Show resolved Hide resolved
src/components/countertop/CountertopWorker.js Outdated Show resolved Hide resolved
src/tools/utils/countertop.js Outdated Show resolved Hide resolved
src/tools/utils/countertop.js Show resolved Hide resolved
src/tools/utils/countertop.js Outdated Show resolved Hide resolved
src/tools/utils/countertop.js Outdated Show resolved Hide resolved
src/tools/utils/countertop.js Outdated Show resolved Hide resolved
src/tools/utils/countertop.js Outdated Show resolved Hide resolved
src/tools/utils/countertop.js Outdated Show resolved Hide resolved
src/tools/utils/countertop.js Outdated Show resolved Hide resolved
@slifty slifty force-pushed the 30-create-stubbed-sous-chef branch 2 times, most recently from 3c10a12 to bb87a6a Compare August 12, 2020 15:17
Yarn moved things around on us, and I'm just going to smile and nod on
this one.
As we start to instantiate appliances there will be several concepts in
the countertop that need an ID to make it easy for mere mortals to know
which copies of appliances / streams exist, and the path of a payload.

Issue #23
@slifty slifty force-pushed the 30-create-stubbed-sous-chef branch from bb87a6a to 179fee0 Compare August 12, 2020 18:17
@slifty slifty changed the title (WIP) Create the Countertop Create countertop topology generation Aug 12, 2020
The countertop is going to need to check Appliances to make sure they
are IAppliances, and various tests will require the creation of
IAppliances and mocks.

Issue #23
These are two little general purpose utilities.  The first will provide
syntactic sugar for array sorts.  The second is an array comparison
helper.
@slifty slifty force-pushed the 30-create-stubbed-sous-chef branch from 179fee0 to a3522e0 Compare August 12, 2020 18:51
Copy link
Contributor

@chriszs chriszs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll admit to still not fully understanding parts of this, but I suppose if it's doing what it's supposed to, then I don't have to.

*
* @return {CountertopStream[]} The array of tributary streams.
*/
getTributaryArray = () => this.mouth.getInputTypes()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this just use .values()? Do we need both types of methods?

Copy link
Member Author

@slifty slifty Aug 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if maybe I didn't know about .values when I wrote this.

I'll go through and see (either way yes I think you're right, and we don't need this method at all thanks to values).

If I remove getTributaryArray then I would replace getTributaryMap with getTributaries

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing: .values returns an iterator (not an array).

Should we have a helper method built into the class, or should folks have to just write Array.from(stream.getTributaries().values()) when they want the array.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That does seem wack.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll keep the method for now, but will implement it using values / the snippet above instead of the .each()

@slifty slifty force-pushed the 30-create-stubbed-sous-chef branch 4 times, most recently from 4b6e6ce to 8bd3ee6 Compare August 17, 2020 19:34
@slifty
Copy link
Member Author

slifty commented Aug 17, 2020

All right -- I am going to merge this despite not reaching perfect self-documented clarity because:

  1. It may change fundamentally some day anyway
  2. It does have an appropriate level of documentation (e.g. all methods are documented), and explaining the overall algorithm may be better suited for the website documentation page.

This adds everything required to generate countertop topologies, where
a toplogy represents the network of data streams that make up the actual
byproducts of the TV Kitchen.

This commit introduces some core concepts:

1. CountertopCoordinator, which is the primary entry point of the
   countertop for developers who are creating a TV Kitchen instance.

2. CountertopStation, which houses a single type of configured
   appliance. A given station might have more than one active copy of
   that appliance, in order to support more than one distinct data
   stream.

3. CountertopStream, which is a thread of data that is being processed
   and tranformed by various appliances / stations.

There will be additional countertop concepts (e.g. the worker) but the
scope of this commit is simply to allow someone to register appliances
with the countertop and have the countertop generate the toplogy of
those appliances (streams and stations).

Topologies are basically a graph where vertices are stations, and edges
form paths which are captured by Streams. We store one stream per
cumulative path (so A, and A => B, A => B => C would each get a distinct
stream).

See issue #23 for a discussion on various topology examples that this
algorithm would generate and support.

Issue #23
Issue #30
Issue #82
Issue #6
@slifty slifty force-pushed the 30-create-stubbed-sous-chef branch from 8bd3ee6 to e22f083 Compare August 17, 2020 19:36
@slifty slifty merged commit 33f1a82 into master Aug 17, 2020
@slifty slifty deleted the 30-create-stubbed-sous-chef branch August 17, 2020 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Define the countertop API How should appliances be installed?
2 participants