Add more "well-known" Sources

[nrbgt]

Elevator Pitch

Expose more DataFrameSource subclasses that know how to work with different files/local features.

Motivation

Inspecting the current environment and local files is good for delivering novel data, at scale, in a way that is relevant to the user's current environment and tools. The WidgetSource added in #35 demonstrates some useful patterns, and could be extended in useful ways, such as toggling features to include in the graph.

Design Ideas

Potential Sources

• PythonSource([imported_module, "module_name"])
  • use the built-in features of e.g. importlib.metadata to discover the relationships between packages and have interesting features like versioned dependencies, named entry_points and other useful features
• CondaSource(["package-name"])
  • be able to look at as-installed on-disk JSON information to gather package dependencies, entry points and other
  • this would not work in lite
• SphinxInventorySource(["path/to/objects.inv])
  • show the relationships of documented symbols to documentation files
• LabPluginsSource(["plugin:id"])
  • show the relationships between extensions, plugins, settings, signals from the live running jupyterlab-derived site
  • this would be difficult to unit test, but would work in lite
• DoitSource("path/to/dodo.py", ["task", "path/to/target.file"])
  • show the task, file and other relationships between doit tasks
    • this is possible in lite, but still pretty hard, as the doit task loader pattern is... complex

Common API

These would all have some common patterns, which would be useful to extract into a common BaseSource, which would become the superclass of DataFrameSource (where most would be no-op`) and these new sources:

• BaseSource.value: TypedTuple(Any())
  • use the upstream convention of a predictable value
    • some sources would accept tuples ofPath/string/JSON
    • others might accept "live" python objects
• BaseSource.node_features: Enum and CustomSource.link_features: Enum
  • provide a common way to mark features which can be discovered in the graph
  • these would become the type field in the node and link data
• BaseSource.find_graph_data()
  • top-level common entry, which would handle processing
    • potentially offer an alternate async entry point, if data will be expensive to discover/parse
• BaseSource.get_default_behaviors(self) -> Tuple[Behavior]
  • return a tuple of behaviors that would get installed by any graph that adds them
  • this would allow for "nice" views, as the data set would know its own column names, etc.
  • probably make this opt-in with BaseSource.add_default_behaviors = True

Out of scope

• adding default ui for all these things
  • the value of this rapidly diminishes as it rapidly increases complexity, and is often still not what a user wants
  • providing more examples like Create example app #7 is a better path forward

[sanbales]

package.json source would be too much of a pain to parse? I barely looked at it a while back and my neophyte gut feeling was that it would be a non-trivial lift to write a parser for it...

[nrbgt]

Sure, yarn.lock is a fine place to stand, and indeed, has an upstream example.

[sanbales]

Yeah, yarn.lock would be much more interesting, the issue I ran into was parsing it with python, found pyarn but it looks a little green (although it is by those RedHat guys so probably pretty good stuff). The bigger issue was adding another dependency, specially one without a conda package.