code reuse?

[thedavidmeister]

this intentionally is a super open ticket, for tracking design discussion around code re-use

"Big ideas"

• code re-use is important
• deterministic code re-use is hard
• UX is critical to network effects

Details

• managed/automated code re-use mechanisms are necessary
  • building complex systems from well tested components (e.g. "don't roll your own crypto")
  • growing an ecosystem of developers through a great developer experience (DX)
  • HC core can provide a rich API of utility fns for common validation requirements, but cannot ever promise to cover all possible use-cases
• bridging agent centric apps facilitates a network of "small" applications
  • removal of monolithic data store
  • encourage API/message passing between agents similar to "micro service"/FaaS architectures
  • bridging promises to remove some, but not all motivations behind code re-use through dependencies
  • how small is "small"?
    • "large" in a traditional system built on package management can easily be several million LOC
    • a "small" app from a traditional system's perspective may still be OOMs too large to manage by hand and maintain a great developer experience
    • it's hard to enumerate what functionality might be needed by many apps at this stage of HC development as there are no large-scale production deployments "in the wild" to reference
    • some illustrative examples based on traditional reasons to re-use code, even in a "micro service" context: cryptography, data type parsing/formatting/coercing/processing, datetime handling, i18n/l10n, logging, URL manipulation, data compression, string manipulation, math, colour theory, logic programming, backports/polyfills, unit conversions, functional programming and other "high level" toolkits for low level langs, machine learning algos, SDKs/APIs, drivers for hardware (e.g. temperature sensor)
• security/hashing of nested dependencies must be treated carefully
  • if we only have a single "top level" hash, we cannot automatically verify the integrity of dependencies, i.e. we cannot use our hash to differentiate between changes in "our" code vs. "upstream" code
  • it is common in many existing package management systems to allow developers to push multiple versions of code under a single semantic version
    • this totally breaks our ability to tie package manager config files to hashes of the overall codebase
    • in this setup, rebuilding/compiling the same source code tomorrow can result in a new hash
  • many package managers make no guarantees at all around determinism, even "deterministic" package managers often won't produce the same hash in reality, or require imposition of additional limits/config/etc. to achieve determinism such as "lock files" (https://yarnpkg.com/blog/2017/05/31/determinism/ doesn't treat "deterministic" as "the same hash" if you consider upstream changes per semantic version, etc.)
• user experience/support/performance of the app store is crucial
  • we cannot expect users to do anything other than click a button to start using an app
  • if we have shared dependencies across many apps, users should not be forced to re-download the shared components of the codebase for every app, if two dependencies have the same hash they should ideally be re-used in situ (e.g. symlink or similar)
  • we cannot expect apps to be built/compiled on the user's device, apps should be prebuilt and simply shipped to the device as needed
  • shipped apps must work across as many devices as possible
• there are "meta problems" around the determinism of the things managing the determinism of the code itself
  • e.g. yarn, that bills itself as a deterministic package manager (see above) is only deterministic if everyone uses the same version of yarn, so do we now need management for our management?
  • hashing files containing configuration for the host system without parsing them and implementing hashing rules for the parsed data is problematic, e.g. the hash of a dna should be based on parsed key/value pairs rather than raw file #654
  • the underlying APIs of HC itself are changing over time, we have a "minimum version" but is this sufficient through major rewrites of HC and/or deprecated/insecure functionality in HC?
  • updates to compilers (e.g. i see people currently recommending babel for es6 support) can result in new hashes for the same source code over time
• where does the code for apps come from when a user wants to download it?
  • we want to be able to distribute and pull all our apps/dependencies from an HC DHT rather than have an external service serving up packages
  • deps should be coming from peers fully built and verifiable
  • we should be able to use the same security mechanisms for data in an app on the app data itself (e.g. code is data)
  • are dependencies downloaded eagerly or lazily? (e.g. if i bridge to an app do i need to recursively pull down all that app's bridges, etc. up front?)
  • the source of code for all HC apps must be an HC app

Approach A: HC core in Rust + Web Assembly

... TODO ...

Approach B: Nix builds

There's a nice slidedeck from @gavinrogers:

https://gavinrogers.github.io/gavinrogers/holochain-package-management.html

[zippy]

moved to: holochain/org#38