Code Reuse Discussion

[zippy]

child of #6

this intentionally is a super open ticket, for tracking design discussion around code re-use

"Big ideas"

• code re-use is important
• deterministic code re-use is hard
• UX and DX is critical to network effects

Details

• managed/automated code re-use mechanisms are necessary
  • building complex systems from well tested components (e.g. "don't roll your own crypto")
  • growing an ecosystem of developers through a great developer experience (DX)
  • HC core can provide a rich API of utility fns for common validation requirements, but cannot ever promise to cover all possible use-cases
• bridging agent centric apps facilitates a network of "small" applications
  • removal of monolithic data store
  • encourage API/message passing between agents similar to "micro service"/FaaS architectures
  • bridging promises to remove some, but not all motivations behind code re-use through dependencies
  • how small is "small"?
    • "large" in a traditional system built on package management can easily be several million LOC
    • a "small" app from a traditional system's perspective may still be OOMs too large to manage by hand and maintain a great developer experience
    • it's hard to enumerate what functionality might be needed by many apps at this stage of HC development as there are no large-scale production deployments "in the wild" to reference
    • some illustrative examples based on traditional reasons to re-use code, even in a "micro service" context: cryptography, data type parsing/formatting/coercing/processing, datetime handling, i18n/l10n, logging, URL manipulation, data compression, string manipulation, math, colour theory, logic programming, backports/polyfills, unit conversions, functional programming and other "high level" toolkits for low level langs, machine learning algos, SDKs/APIs, drivers for hardware (e.g. temperature sensor)
• security/hashing of nested dependencies must be treated carefully
  • if we only have a single "top level" hash, we cannot automatically verify the integrity of dependencies, i.e. we cannot use our hash to differentiate between changes in "our" code vs. "upstream" code
  • it is common in many existing package management systems to allow developers to push multiple versions of code under a single semantic version
    • this totally breaks our ability to tie package manager config files to hashes of the overall codebase
    • in this setup, rebuilding/compiling the same source code tomorrow can result in a new hash
  • many package managers make no guarantees at all around determinism, even "deterministic" package managers often won't produce the same hash in reality, or require imposition of additional limits/config/etc. to achieve determinism such as "lock files" (https://yarnpkg.com/blog/2017/05/31/determinism/ doesn't treat "deterministic" as "the same hash" if you consider upstream changes per semantic version, etc.)
• user experience/support/performance of the app store is crucial
  • we cannot expect users to do anything other than click a button to start using an app
  • if we have shared dependencies across many apps, users should not be forced to re-download the shared components of the codebase for every app, if two dependencies have the same hash they should ideally be re-used in situ (e.g. symlink or similar)
  • we cannot expect apps to be built/compiled on the user's device, apps should be prebuilt and simply shipped to the device as needed
  • shipped apps must work across as many devices as possible
• there are "meta problems" around the determinism of the things managing the determinism of the code itself
  • e.g. yarn, that bills itself as a deterministic package manager (see above) is only deterministic if everyone uses the same version of yarn, so do we now need management for our management?
  • hashing files containing configuration for the host system without parsing them and implementing hashing rules for the parsed data is problematic, e.g. #654
  • the underlying APIs of HC itself are changing over time, we have a "minimum version" but is this sufficient through major rewrites of HC and/or deprecated/insecure functionality in HC?
  • updates to compilers (e.g. i see people currently recommending babel for es6 support) can result in new hashes for the same source code over time
• where does the code for apps come from when a user wants to download it?
  • we want to be able to distribute and pull all our apps/dependencies from an HC DHT rather than have an external service serving up packages
  • deps should be coming from peers fully built and verifiable
  • we should be able to use the same security mechanisms for data in an app on the app data itself (e.g. code is data)
  • are dependencies downloaded eagerly or lazily? (e.g. if i bridge to an app do i need to recursively pull down all that app's bridges, etc. up front?)
  • the source of code for all HC apps must be an HC app

Approach A: HC core in Rust + Web Assembly

... TODO ...

Approach B: Nix builds

There's a nice slidedeck from @gavinrogers:

https://gavinrogers.github.io/gavinrogers/holochain-package-management.html

[ddd-mtl]

From what I understand, the main questions is "How do we handle code dependency?"
And breaks down into different problems:

1. Is it possible for two different apps to share a same library and do some kind of runtime optimization for that?
2. How can an end-user trust a new or updated app?
3. Can app devs leverage code from other devs in there apps?

What is confusing to me is the discussion seems to mix all different levels of 'apps':

1. Web app: An app that uses one or several HC apps plus extra glue & custom code
2. HC app: Nucleus + UI
3. Nucleus: DNA of one or several Zomes

(Then there is also mixins and 'middleware' for DNA or UI part of an HC app.)

My feeling is these different levels have different answers to the main question.