Major refactor of pipeline system to support next wave of features.

All assets which are to be processed by trunk must now be declared as HTML `link` elements as such: `<link data-trunk rel="rust|sass|..." data-attr0 data-attr1/>`. The links may appear anywhere in the HTML and Trunk will process them and replace them or delete them based on the associated pipeline's output. If the link element does not have the `data-trunk` attribute, it will not be processed. The `--manifest-path` flag has been removed in favor of the new link style: `<link rel="rust" href="path/to/Cargo.toml"/>`. This is optional, and will default to the Cargo.toml of the source HTML's parent dir if not specified. Update README describing new asset pipeline declaration pattern. closes #50
trunk-rs · Oct 6, 2020 · e6c23e1 · e6c23e1
1 parent 212e901
commit e6c23e1
Show file tree

Hide file tree

Showing 29 changed files with 1,254 additions and 795 deletions.
diff --git a/.github/workflows/release.yaml b/.github/workflows/release.yaml
@@ -116,6 +116,6 @@ jobs:
         with:
           files: trunk-*/trunk-*
           body_path: RELEASE_LOG.md
-          prerelease: true
+          draft: true
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,6 +3,16 @@ changelog
 This changelog follows the patterns described here: https://keepachangelog.com/en/1.0.0/.
 
 ## Unreleased
+- Made a lot of refactoring progress to be able to support [#28](https://github.com/thedodd/trunk/issues/28) & [#46](https://github.com/thedodd/trunk/issues/46). More work remains to be done, but the foundation to be able to knock these items out is now in place.
+
+### changed
+- All assets which are to be processed by trunk must now be declared as HTML `link` elements as such: `<link data-trunk rel="rust|sass|css|icon|copy-file|..." data-attr0 data-attr1/>`. The links may appear anywhere in the HTML and Trunk will process them and replace them or delete them based on the associated pipeline's output. If the link element does not have the `data-trunk` attribute, it will not be processed.
+
+### fixed
+- Fixed [#50](https://github.com/thedodd/trunk/issues/50): the ability to copy a file or an entire dir into the dist dir is now supported with two different pipeline types: `<link data-trunk rel="copy-file" href="target/file"/>` and `<link data-trunk rel="copy-dir" href="target/dir"/>` respectively.
+
+### removed
+- The `manifest-path` option has been removed from all Trunk subcommands. The path to the `Cargo.toml` is now specified in the source HTML as `<link rel="rust" href="path/to/Cargo.toml"/>`. The `href="..."` attribute may be omitted, in which case Trunk will look for a `Cargo.toml` within the same directory as the source HTML. If the `href` attribute points to a directory, Trunk will look for the `Cargo.toml` file in that directory.
 
 ## 0.6.0
 ### added

diff --git a/Cargo.lock b/Cargo.lock
diff --git a/Cargo.toml b/Cargo.toml
@@ -1,6 +1,6 @@
 [package]
 name = "trunk"
-version = "0.6.0"
+version = "0.7.0"
 edition = "2018"
 description = "Build, bundle & ship your Rust WASM application to the web."
 license = "MIT/Apache-2.0"
@@ -34,7 +34,7 @@ seahash = "4.0.1"
 serde = { version="1", features=["derive"] }
 structopt = "0.3.18"
 structopt-derive = "0.4.11"
-surf = "=2.0.0-alpha.7"
+surf = "2.0.0"
 tide = { version="0.13.0", features=["unstable"] }
 toml = "0.5.6"
 

diff --git a/README.md b/README.md
@@ -23,16 +23,15 @@ Trunk is a WASM web application bundler for Rust. Trunk uses a simple, zero-conf
 ### install
 First, install Trunk via one of the following options.
 ```bash
-# Install via cargo.
-cargo install trunk
+# Install via homebrew on Mac, Linux or Windows (WSL).
+brew install trunk
 
 # Install a release binary (great for CI).
 VERSION=v0.6.0
 wget -qO- https://github.com/thedodd/trunk/releases/download/${VERSION}/trunk-x86_64-unknown-linux-gnu.tar.gz | tar -xzf-
 
-# Install via homebrew on Mac, Linux or Windows (WSL).
-# (coming soon)
-brew install trunk
+# Install via cargo.
+cargo install trunk
 ```
 <small>Release binaries can be found on the [Github releases page](https://github.com/thedodd/trunk/releases).</small>
 
@@ -55,7 +54,7 @@ Trunk uses a source HTML file to drive all asset building and bundling. Trunk al
 ```html
 <html>
   <head>
-    <link rel="stylesheet" href="index.scss"/>
+    <link data-trunk rel="scss" href="path/to/index.scss"/>
   </head>
 </html>
 ```
@@ -95,19 +94,36 @@ Trunk leverages Rust's powerful concurrency primitives for maximum build speeds
 `trunk config show` prints out Trunk's current config, before factoring in CLI arguments. Nice for testing & debugging.
 
 ## assets
+Declaring assets to be processed by Trunk is simple and extensible. All assets to be processed by Trunk must follow these three rules:
+- must be declared as a valid HTML `link` tag.
+- must have the attribute `data-trunk`.
+- must have the attribute `rel="{type}"`, where `{type}` is one of the asset types listed below.
+
+This will typically look like: `<link data-trunk rel="{type}" href="{path}" ..other options here.. />`. Each asset type described below specifies the required and optional attributes for its asset type. All `<link data-trunk .../>` HTML elements will be replaced with the output HTML of the associated pipeline.
+
+Currently supported asset types:
+- ✅ `rust`: Trunk will compile the specified Cargo project as the main WASM application. This is optional. If not specified, Trunk will look for a `Cargo.toml` in the parent directory of the source HTML file.
+  - `href`: (optional) the path to the `Cargo.toml` of the Rust project. If a directory is specified, then Trunk will look for the `Cargo.toml` in the given directory. If no value is specified, then Trunk will look for a `Cargo.toml` in the parent directory of the source HTML file.
+  - `data-bin`: (optional) the name of the binary to compile and use as the main WASM application. If the Cargo project has multiple binaries, this value will be required for proper functionality.
+- ✅ `sass`, `scss`: Trunk ships with a [built-in sass/scss compiler](https://github.com/compass-rs/sass-rs). Just link to your sass files from your source HTML, and Trunk will handle the rest. This content is hashed for cache control. The `href` attribute must be included in the link pointing to the sass/scss file to be processed.
+- ✅ `css`: Trunk will copy linked css files found in the source HTML without content modification. This content is hashed for cache control. The `href` attribute must be included in the link pointing to the css file to be processed.
+  - In the future, Trunk will resolve local `@imports`, will handle minification (see [trunk#7](https://github.com/thedodd/trunk/issues/3)), and we may even look into a pattern where any CSS found in the source tree will be bundled, which would enable a nice zero-config "component styles" pattern. See [trunk#3](https://github.com/thedodd/trunk/issues/3) for more details.
+- ✅ `icon`: Trunk will copy the icon image specified in the `href` attribute to the `dist` dir. This content is hashed for cache control.
+- ✅ `copy-file`: Trunk will copy the file specified in the `href` attribute to the `dist` dir. This content is copied exactly, no hashing is performed.
+- ✅ `copy-dir`: Trunk will recursively copy the directory specified in the `href` attribute to the `dist` dir. This content is copied exactly, no hashing is performed.
+- ⏳ `rust-worker`: (in-progress) Trunk will compile the specified Rust project as a WASM web worker. The following attributes are required:
+  - `href`: (optional) the path to the `Cargo.toml` of the Rust project. If a directory is specified, then Trunk will look for the `Cargo.toml` in the given directory. If no value is specified, then Trunk will look for a `Cargo.toml` in the parent directory of the source HTML file.
+  - `data-bin`: (optional) the name of the binary to compile and use as the web worker. If the Cargo project has multiple binaries, this value will be required for proper functionality.
+
 Trunk is still a young project, and new asset types will be added as we move forward. Keep an eye on [trunk#3](https://github.com/thedodd/trunk/issues/3) for more information on planned asset types, implementation status, and please contribute to the discussion if you think something is missing.
 
-Currently supported assets:
-- ✅ `sass`: Trunk ships with a [built-in sass/scss compiler](https://github.com/compass-rs/sass-rs). Just link to your sass files from your source HTML, and Trunk will handle the rest. This content is hashed for cache control.
-- ✅ `css`: Trunk will copy linked css files found in the source HTML without content modification. This content is hashed for cache control.
-  - In the future, Trunk will resolve local `@imports`, will handle minification (see [trunk#7](https://github.com/thedodd/trunk/issues/3)), and we may even look into a pattern where any CSS found in the source tree will be bundled, which would enable a nice zero-config "component styles" pattern. See [trunk#3](https://github.com/thedodd/trunk/issues/3) for more details.
-- ✅ `icon`: Trunk will automatically copy referenced icons to the `dist` dir. This content is hashed for cache control.
-- ✅ `js snippets`: [wasm-bindgen JS snippets](https://rustwasm.github.io/docs/wasm-bindgen/reference/js-snippets.html) are automatically copied to the dist dir, hashed and ready to rock.
+### js snippets
+JS snippets generated from the [wasm-bindgen JS snippets feature](https://rustwasm.github.io/docs/wasm-bindgen/reference/js-snippets.html) are automatically copied to the dist dir, hashed and ready to rock. No additional setup is required. Just use the feature in your application, and Trunk will take care of the rest.
 
 ### images & other resources
-Images and other resource types can be copied into the `dist` dir by adding a link like this to your source HTML: `<link rel="trunk-dist" href="path/to/resource"/>` (note the `rel="trunk-dist"` attribute). This will cause Trunk to find the target resource, and copy it to the `dist` dir unmodified. No hashing will be applied. The link itself will be removed from the HTML.
+Images and other resource types can be copied into the `dist` dir by adding a link like this to your source HTML: `<link data-trunk rel="copy-file" href="path/to/image"/>`. Any normal file type is supported. This will cause Trunk to find the target resource, and copy it to the `dist` dir unmodified. No hashing will be applied. The link itself will be removed from the HTML. To copy an entire directory of assets/images, you can use the following HTML: `<link data-trunk rel="copy-dir" href="path/to/images-dir"/>`.
 
-This will allow your WASM application to reference images directly from the `dist` dir, and Trunk will ensure that the images are available in the `dist` dir to be served. You will need to be sure to use the correct public URL in your code, which can be configured via the `--public-url` CLI flag to Trunk.
+This will allow your WASM application to reference images directly from the `dist` dir, and Trunk will ensure that the images are available in the `dist` dir to be served.
 
 **NOTE:** as Trunk continues to mature, we will find better ways to include images and other resources. Hashing content for cache control is great, we just need to find a nice pattern to work with images referenced in Rust components. Please contribute to the discussion over in [trunk#9](https://github.com/thedodd/trunk/issues/9)! See you there.
 

diff --git a/Trunk.toml b/Trunk.toml
@@ -7,8 +7,6 @@ target = "index.html"
 release = false
 # The output dir for all final assets.
 dist = "dist"
-# Path to Cargo.toml.
-manifest = "Cargo.toml"
 # The public URL from which assets are to be served.
 public_url = "/"
 

diff --git a/examples/yew/Trunk.toml b/examples/yew/Trunk.toml
@@ -1,4 +1,3 @@
 [build]
 target = "index.html"
 dist = "dist"
-manifest = "Cargo.toml"
diff --git a/examples/yew/index.html b/examples/yew/index.html
@@ -5,12 +5,14 @@
     <meta name="viewport" content="width=device-width, initial-scale=1"/>
     <title>Trunk | Yew | YBC</title>
     <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bulma@0.9.0/css/bulma.min.css"/>
-    <link rel="stylesheet" href="src/index.scss"/>
-    <link rel="stylesheet" href="src/app.css"/>
-    <link rel="icon" href="src/yew.svg"/>
-    <link rel="trunk-dist" href="src/yew.svg"/>
+
+    <link data-trunk rel="scss" href="src/index.scss"/>
+    <link data-trunk rel="css" href="src/app.css"/>
+    <link data-trunk rel="icon" href="src/yew.svg"/>
+    <link data-trunk rel="copy-file" href="src/yew.svg"/>
     <base data-trunk-public-url/>
 </head>
 <body>
+    <link data-trunk rel="rust" href="Cargo.toml" data-bin="yew-example"/>
 </body>
 </html>
diff --git a/examples/yew/src/main.rs b/examples/yew/src/main.rs
@@ -27,7 +27,7 @@ impl Component for App {
     }
 
     fn view(&self) -> Html {
-        html!{
+        html! {
             <>
             <ybc::Navbar
                 classes="is-success"
@@ -97,7 +97,6 @@ impl Component for App {
     }
 }
 
-
 #[wasm_bindgen(inline_js = "export function snippetTest() { console.log('Hello from JS FFI!'); }")]
 extern "C" {
     fn snippetTest();

diff --git a/src/build.rs b/src/build.rs
@@ -1,16 +1,16 @@
 //! Build system & asset pipelines.
 
+use std::path::PathBuf;
 use std::sync::Arc;
 
 use anyhow::Result;
 use async_std::fs;
+use futures::channel::mpsc::Sender;
 use indicatif::ProgressBar;
 
 use crate::common::{BUILDING, ERROR, SUCCESS};
 use crate::config::RtcBuild;
-use crate::pipelines::cargo::CargoBuild;
-use crate::pipelines::html::HtmlPipeline;
-use crate::pipelines::wasmbg::WasmBindgen;
+use crate::pipelines::HtmlPipeline;
 
 /// A system used for building a Rust WASM app & bundling its assets.
 ///
@@ -21,11 +21,7 @@ use crate::pipelines::wasmbg::WasmBindgen;
 pub struct BuildSystem {
     /// Runtime config.
     cfg: Arc<RtcBuild>,
-    /// Cargo build pipeline system.
-    cargo_build_pipeline: Arc<CargoBuild>,
-    /// WASM bindgen build pipeline system.
-    wasm_bindgen_pipeline: Arc<WasmBindgen>,
-    /// HTML build pipeline system.
+    /// HTML build pipeline.
     html_pipeline: Arc<HtmlPipeline>,
     /// The build system progress bar for displaying the state of the build system overall.
     progress: ProgressBar,
@@ -36,18 +32,10 @@ impl BuildSystem {
     ///
     /// Reducing the number of assumptions here should help us to stay flexible when adding new
     /// commands, rafctoring and the like.
-    pub async fn new(cfg: Arc<RtcBuild>, progress: ProgressBar) -> Result<Self> {
-        let mode_segment = if cfg.release { "release" } else { "debug" };
-        let bindgen_out = Arc::new(cfg.manifest.metadata.target_directory.join("wasm-bindgen").join(mode_segment));
-
-        let cargo_build_pipeline = Arc::new(CargoBuild::new(cfg.clone(), progress.clone()));
-        let wasm_bindgen_pipeline = Arc::new(WasmBindgen::new(cfg.clone(), bindgen_out, progress.clone()));
-        let html_pipeline = Arc::new(HtmlPipeline::new(cfg.clone(), progress.clone())?);
-
+    pub async fn new(cfg: Arc<RtcBuild>, progress: ProgressBar, ignore_chan: Option<Sender<PathBuf>>) -> Result<Self> {
+        let html_pipeline = Arc::new(HtmlPipeline::new(cfg.clone(), progress.clone(), ignore_chan)?);
         Ok(Self {
             cfg,
-            cargo_build_pipeline,
-            wasm_bindgen_pipeline,
             html_pipeline,
             progress,
         })
@@ -71,26 +59,20 @@ impl BuildSystem {
             Err(err) => {
                 self.progress.set_prefix(&format!("{}", ERROR));
                 self.progress.finish_with_message("error");
-                self.progress.println(err.to_string());
                 Err(err)
             }
         }
     }
 
     async fn do_build(&mut self) -> Result<()> {
-        // Spawn cargo build. It will run concurrently without polling.
-        let cargo_handle = self.cargo_build_pipeline.clone().spawn();
+        // TODO: delete the contents of the `dist/.current` dir (currently in flight elsewhere).
 
         // Ensure the output dist directory is in place.
         fs::create_dir_all(self.cfg.dist.as_path()).await?;
 
-        // Spawn the wasm-bindgen call, it will await the cargo build.
-        let wasmbg_handle = self.wasm_bindgen_pipeline.clone().spawn(cargo_handle);
-
-        // Spawn the source HTML pipeline. This will spawn all other asset pipelines derived from
-        // the source HTML, and will await the cargo build & wasm-bindgen build in order to
-        // generate the final HTML.
-        self.html_pipeline.clone().spawn(wasmbg_handle).await?;
+        // Spawn the source HTML pipeline. This will spawn all other pipelines derived from
+        // the source HTML, and will ultimately generate and write the final HTML.
+        self.html_pipeline.clone().spawn().await?;
         Ok(())
     }
 }
diff --git a/src/cmd/build.rs b/src/cmd/build.rs
@@ -18,7 +18,7 @@ pub struct Build {
 impl Build {
     pub async fn run(self, config: Option<PathBuf>) -> Result<()> {
         let cfg = ConfigOpts::rtc_build(self.build, config).await?;
-        let mut system = BuildSystem::new(cfg, spinner()).await?;
+        let mut system = BuildSystem::new(cfg, spinner(), None).await?;
         system.build().await?;
         Ok(())
     }

diff --git a/src/cmd/clean.rs b/src/cmd/clean.rs
@@ -1,6 +1,6 @@
 use std::path::PathBuf;
 
-use anyhow::Result;
+use anyhow::{ensure, Result};
 use async_process::{Command, Stdio};
 use async_std::fs;
 use structopt::StructOpt;
@@ -26,9 +26,7 @@ impl Clean {
                 .stderr(Stdio::piped())
                 .output()
                 .await?;
-            if !output.status.success() {
-                eprintln!("{}", String::from_utf8_lossy(&output.stderr));
-            }
+            ensure!(output.status.success(), "{}", String::from_utf8_lossy(&output.stderr));
         }
         Ok(())
     }

diff --git a/src/common.rs b/src/common.rs
@@ -2,7 +2,7 @@
 
 use std::path::PathBuf;
 
-use anyhow::{Context, Result};
+use anyhow::{anyhow, Context, Result};
 use async_std::path::PathBuf as AsyncPathBuf;
 use async_std::task::spawn_blocking;
 
@@ -24,7 +24,7 @@ pub fn parse_public_url(val: &str) -> String {
 /// A utility function to recursively copy a directory.
 pub async fn copy_dir_recursive(from_dir: PathBuf, to_dir: PathBuf) -> Result<()> {
     if !AsyncPathBuf::from(&from_dir).exists().await {
-        return Ok(());
+        return Err(anyhow!("directory can not be copied as it does not exist {:?}", &from_dir));
     }
     spawn_blocking(move || {
         let opts = fs_extra::dir::CopyOptions {

diff --git a/src/config/manifest.rs b/src/config/manifest.rs
@@ -1,4 +1,4 @@
-use std::path::PathBuf;
+use std::path::Path;
 
 use anyhow::{anyhow, Result};
 use async_std::task::spawn_blocking;
@@ -16,16 +16,11 @@ pub struct CargoMetadata {
 }
 
 impl CargoMetadata {
-    /// Get the project's cargo metadata of the CWD, or of the project specified by the given manifest path.
-    pub async fn new(manifest: &Option<PathBuf>) -> Result<Self> {
-        // Fetch the cargo project's metadata.
+    // Create a new instance from the Cargo.toml at the given path.
+    pub async fn new(manifest: &Path) -> Result<Self> {
         let mut cmd = MetadataCommand::new();
-        if let Some(manifest) = manifest.as_ref() {
-            cmd.manifest_path(manifest);
-        }
+        cmd.manifest_path(manifest);
         let metadata = spawn_blocking(move || cmd.exec()).await?;
-
-        // Get a handle to this project's package info.
         let package = metadata
             .root_package()
             .cloned()