Public review draft tech arch vision

[WesIngwersen]

This PR serves as a public review of the Technical Architecture Vision paper. View the draft vision in a legible format.

To comment on specific line:
Look under Files changed and the Cornerstone_Architecture_Vision.md and then click the + symbol on the line of your choice to add a comment.

For general comments, just add them below.

[staadecker]

Hi Wes,

Not sure if this is the right place to provide feedback however I thought I'd ask what Python libraries Cornerstone was planning to use in its data pipelines? Specifically, I know pandas is often used, however, I would strongly recommend Polars as an alternative library for manipulating DataFrames.

Polars is much faster than pandas since its backend is written in Rust and it multi-threads all computations by splitting DataFrames into chunks that are each processed by a different core. (It also has a query engine that can simplify chained queries). Most importantly, Polars' syntax is extremely legible and powerful (no pandas indexes to mess with or in-place operations). I've come to love Polars over pandas and simply thought I'd let the team know :)

[staadecker]

Otherwise, I think Python, monorepo, and ETL pipeline are excellent calls! I also love the .yaml based approach that I'm seeing in the bedrock repo.

[WesIngwersen]

Hi Wes,

Not sure if this is the right place to provide feedback however I thought I'd ask what Python libraries Cornerstone was planning to use in its data pipelines? Specifically, I know pandas is often used, however, I would strongly recommend Polars as an alternative library for manipulating DataFrames.

Polars is much faster than pandas since its backend is written in Rust and it multi-threads all computations by splitting DataFrames into chunks that are each processed by a different core. (It also has a query engine that can simplify chained queries). Most importantly, Polars' syntax is extremely legible and powerful (no pandas indexes to mess with or in-place operations). I've come to love Polars over pandas and simply thought I'd let the team know :)

Thanks @staadecker for this comment. We don't get into specifics of supporting Python packages in this paper but we are looking into drawing on this to speed up handling of large matrices. Great suggestion.