Added support for virtual projects (#551)

CHANGELOG.md

@@ -10,6 +10,9 @@ _Unreleased_
 - Improved the error message when an update could not be performed because files
   are in use.  #550
 
+- Rye now supports virtual projects.  These are themselves not installed into the
+  virtualenv but their dependencies are.  #551
+
 - Update the Python internals (python external dependencies) to new versions.  #553
 
 - Update to newer versions of pip tools.  For Python 3.7 `6.14.0` is used, for

docs/guide/pyproject.md

@@ -88,6 +88,23 @@ can be forced enabled in the global config.
 managed = true
 ```
 
+## `tool.rye.virtual`
+
++++ 0.20.0
+
+If this key is set to `true` the project is declared as a virtual project.  This is a special
+mode in which the package itself is not installed, but only the dependencies are.  This is
+for instance useful if you are not creating a Python project, but you are depending on Python
+software.  As an example you can use this to install software written in Python.  This key is
+set to true when `rye init` is invoked with the `--virtual` flag.
+
+```toml
+[tool.rye]
+virtual = true
+```
+
+For more information consult the [Virtual Project Guide](../virtual/).
+
 ## `tool.rye.sources`
 
 This is an array of tables with sources that should be used for locating dependencies.
@@ -174,13 +191,15 @@ hello-world = { call = "builtins:print('Hello World!')" }
 
 ## `tool.rye.workspace`
 
-When a table with that key is stored, then a project is declared to be a workspace root.  By
-default all Python projects discovered in sub folders will then become members of this workspace
-and share a virtualenv.  Optionally the `members` key (an array) can be used to restrict these
-members.  In that list globs can be used.  The root project itself is always a member.
+When a table with that key is stored, then a project is declared to be a
+[workspace](../workspaces/) root.  By default all Python projects discovered in
+sub folders will then become members of this workspace and share a virtualenv.
+Optionally the `members` key (an array) can be used to restrict these members.
+In that list globs can be used.  The root project itself is always a member.
 
 ```toml
 [tool.rye.workspace]
 members = ["mylib-*"]
 ```
 
+For more information consult the [Workspaces Guide](../workspaces/).

docs/guide/sync.md

@@ -33,7 +33,7 @@ rye add --optional=web flask
 rye lock --features=web
 ```
 
-When working with workspaces, the package name needs to be prefixed with a slash:
+When working with [workspaces](../workspaces/), the package name needs to be prefixed with a slash:
 
 ```
 rye lock --features=package-name/feature-name

docs/guide/virtual.md

@@ -0,0 +1,32 @@
+# Virtual Projects
+
++++ 0.20.0
+
+Virtual projects are projects which are themselves not installable Python
+packages, but that will sync their dependencies.  They are declared like a
+normal python package in a `pyproject.toml`, but they do not create a package.
+Instead the `tool.rye.virtual` key is set to `true`.
+
+For instance this is useful if you want to use a program like `mkdocs` without
+declaring a package yourself:
+
+```
+rye init --virtual
+rye add mkdocs
+rye sync
+rye run mkdocs
+```
+
+This will create a `pyproject.toml` but does not actually declare any python code itself.
+Yet when synching you will end up with mkdocs in your project.
+
+## Behavior Changes
+
+When synching the project itself is never installed into the virtualenv as it's not
+considered to be a valid package.  Likewise you cannot publish virtual packages to
+PyPI or another index.
+
+## Workspaces
+
+If a [workspace](../workspaces/) does not have a toplevel package it's
+recommended that it's declared as virtual.

docs/guide/workspaces.md

@@ -0,0 +1,48 @@
+# Workspaces
+
+Workspaces are a feature that allows you to work with multiple packages that
+have dependencies to each other.  A workspace is declared by setting the
+`tool.rye.workspace` key a `pyproject.toml`.  Afterwards all projects within
+that workspace share a singular virtualenv.
+
+## Declaring Workspaces
+
+A workspace is declared by the "toplevel" `pyproject.toml`.  At the very least
+the key `tool.rye.workspace` needs to be added.  It's recommended that a glob
+pattern is also set in the `members` key to prevent accidentally including
+unintended folders as projects.
+
+```toml
+[tool.rye.workspace]
+members = ["myname-*"]
+```
+
+This declares a workspace where all folders starting with the name `myname-`
+are considered.  If the toplevel workspace in itself should not be a project,
+then it should be declared as a virtual package:
+
+```toml
+[tool.rye]
+virtual = true
+
+[tool.rye.workspace]
+members = ["myname-*"]
+```
+
+For more information on that see [Virtual Packages](../virtual/).
+
+## Syncing
+
+In a workspace it does not matter which project you are working with, the entire
+workspace is synchronized at all times.  This has some untypical consequences but
+simplifies the general development workflow.
+
+When a package depends on another package it's first located in the workspace locally
+before it's attempted to be downloaded from an index.  The `--all-features` flag is
+automatically applied to all packages, but to turn on the feature of a specific
+package the feature name must be prefixed.  For instance to enable the `foo` extra feature
+of the `myname-bar` package you would need to do this:
+
+```
+rye sync --features=myname-bar/foo
+```

mkdocs.yml

@@ -19,6 +19,8 @@ nav:
     - Rust Modules: guide/rust.md
     - Dependency Sources: guide/sources.md
     - Dependencies: guide/deps.md
+    - Workspaces: guide/workspaces.md
+    - Virtual Projects: guide/virtual.md
     - Toolchains:
       - guide/toolchains/index.md
       - Portable CPython: guide/toolchains/cpython.md

rye/src/cli/build.rs

@@ -96,6 +96,11 @@ pub fn execute(cmd: Args) -> Result<(), Error> {
     }
 
     for project in projects {
+        // skip over virtual packages on build
+        if project.is_virtual() {
+            continue;
+        }
+
         if output != CommandOutput::Quiet {
             echo!("building {}", style(project.normalized_name()?).cyan());
         }

rye/src/cli/init.rs

@@ -62,6 +62,12 @@ pub struct Args {
     /// Don't import from setup.cfg, setup.py, or requirements files.
     #[arg(long)]
     no_import: bool,
+    /// Initialize this as a virtual package.
+    ///
+    /// A virtual package can have dependencies but is itself not installed as a
+    /// Python package.  It also cannot be published.
+    #[arg(long = "virtual")]
+    is_virtual: bool,
     /// Requirements files to initialize pyproject.toml with.
     #[arg(short, long, name = "REQUIREMENTS_FILE", conflicts_with = "no_import")]
     requirements: Option<Vec<PathBuf>>,
@@ -111,6 +117,8 @@ classifiers = ["Private :: Do Not Upload"]
 [project.scripts]
 hello = {{ name_safe ~ ":hello"}}
 
+{%- if not is_virtual %}
+
 [build-system]
 {%- if build_system == "hatchling" %}
 requires = ["hatchling"]
@@ -128,9 +136,13 @@ build-backend = "pdm.backend"
 requires = ["maturin>=1.2,<2.0"]
 build-backend = "maturin"
 {%- endif %}
+{%- endif %}
 
 [tool.rye]
 managed = true
+{%- if is_virtual %}
+virtual = true
+{%- endif %}
 {%- if dev_dependencies %}
 dev-dependencies = [
 {%- for dependency in dev_dependencies %}
@@ -141,6 +153,7 @@ dev-dependencies = [
 dev-dependencies = []
 {%- endif %}
 
+{%- if not is_virtual %}
 {%- if build_system == "hatchling" %}
 
 [tool.hatch.metadata]
@@ -154,7 +167,7 @@ packages = [{{ "src/" ~ name_safe }}]
 python-source = "python"
 module-name = {{ name_safe ~ "._lowlevel" }}
 features = ["pyo3/extension-module"]
-
+{%- endif %}
 {%- endif %}
 
 "#;
@@ -265,6 +278,7 @@ pub fn execute(cmd: Args) -> Result<(), Error> {
     let readme = dir.join("README.md");
     let license_file = dir.join("LICENSE.txt");
     let python_version_file = dir.join(".python-version");
+    let is_virtual = cmd.is_virtual;
 
     if toml.is_file() {
         bail!("pyproject.toml already exists");
@@ -450,52 +464,57 @@ pub fn execute(cmd: Args) -> Result<(), Error> {
             license => metadata.license,
             dependencies => metadata.dependencies,
             dev_dependencies => metadata.dev_dependencies,
+            is_virtual,
             with_readme,
             build_system,
             private,
         },
     )?;
     fs::write(&toml, rv).context("failed to write pyproject.toml")?;
 
-    let src_dir = dir.join("src");
-    if !imported_something && !src_dir.is_dir() {
-        let name = metadata.name.expect("project name");
-        if is_rust {
-            fs::create_dir_all(&src_dir).ok();
-            let project_dir = dir.join("python").join(name.replace('-', "_"));
-            fs::create_dir_all(&project_dir).ok();
-            let rv = env.render_named_str("lib.rs", LIB_RS_TEMPLATE, context! { name })?;
-            fs::write(src_dir.join("lib.rs"), rv).context("failed to write lib.rs")?;
-            let rv = env.render_named_str(
-                "Cargo.json",
-                CARGO_TOML_TEMPLATE,
-                context! {
-                    name,
-                    name_safe,
-                },
-            )?;
-            fs::write(dir.join("Cargo.toml"), rv).context("failed to write Cargo.toml")?;
-            let rv = env.render_named_str(
-                "__init__.py",
-                RUST_INIT_PY_TEMPLATE,
-                context! {
-                    name_safe
-                },
-            )?;
-            fs::write(project_dir.join("__init__.py"), rv)
-                .context("failed to write __init__.py")?;
-        } else {
-            let project_dir = src_dir.join(name.replace('-', "_"));
-            fs::create_dir_all(&project_dir).ok();
-            let rv = env.render_named_str("__init__.py", INIT_PY_TEMPLATE, context! { name })?;
-            fs::write(project_dir.join("__init__.py"), rv)
-                .context("failed to write __init__.py")?;
+    if !is_virtual {
+        let src_dir = dir.join("src");
+        if !imported_something && !src_dir.is_dir() {
+            let name = metadata.name.expect("project name");
+            if is_rust {
+                fs::create_dir_all(&src_dir).ok();
+                let project_dir = dir.join("python").join(name.replace('-', "_"));
+                fs::create_dir_all(&project_dir).ok();
+                let rv = env.render_named_str("lib.rs", LIB_RS_TEMPLATE, context! { name })?;
+                fs::write(src_dir.join("lib.rs"), rv).context("failed to write lib.rs")?;
+                let rv = env.render_named_str(
+                    "Cargo.json",
+                    CARGO_TOML_TEMPLATE,
+                    context! {
+                        name,
+                        name_safe,
+                    },
+                )?;
+                fs::write(dir.join("Cargo.toml"), rv).context("failed to write Cargo.toml")?;
+                let rv = env.render_named_str(
+                    "__init__.py",
+                    RUST_INIT_PY_TEMPLATE,
+                    context! {
+                        name_safe
+                    },
+                )?;
+                fs::write(project_dir.join("__init__.py"), rv)
+                    .context("failed to write __init__.py")?;
+            } else {
+                let project_dir = src_dir.join(name.replace('-', "_"));
+                fs::create_dir_all(&project_dir).ok();
+                let rv =
+                    env.render_named_str("__init__.py", INIT_PY_TEMPLATE, context! { name })?;
+                fs::write(project_dir.join("__init__.py"), rv)
+                    .context("failed to write __init__.py")?;
+            }
         }
     }
 
     echo!(
-        "{} Initialized project in {}",
+        "{} Initialized {}project in {}",
         style("success:").green(),
+        if is_virtual { "virtual " } else { "" },
         dir.display()
     );
     echo!("  Run `rye sync` to get started");

rye/src/cli/publish.rs

@@ -58,6 +58,10 @@ pub fn execute(cmd: Args) -> Result<(), Error> {
     let venv = ensure_self_venv(output)?;
     let project = PyProject::discover()?;
 
+    if project.is_virtual() {
+        bail!("virtual packages cannot be published");
+    }
+
     // Get the files to publish.
     let files = match cmd.dist {
         Some(paths) => paths,

rye/src/cli/show.rs

@@ -46,6 +46,7 @@ pub fn execute(cmd: Args) -> Result<(), Error> {
             }
         }
     }
+    echo!("virtual: {}", style(project.is_virtual()).cyan());
 
     if let Some(workspace) = project.workspace() {
         echo!(

rye/src/lock.rs

@@ -94,7 +94,12 @@ pub fn update_workspace_lockfile(
         let pyproject = pyproject_result?;
         let rel_url = make_relative_url(&pyproject.root_path(), &workspace.path())?;
         let applicable_extras = format_project_extras(features_by_project.as_ref(), &pyproject)?;
-        writeln!(local_req_file, "-e {}{}", rel_url, applicable_extras)?;
+
+        // virtual packages are not installed
+        if !pyproject.is_virtual() {
+            writeln!(local_req_file, "-e {}{}", rel_url, applicable_extras)?;
+        }
+
         local_projects.insert(pyproject.normalized_name()?, rel_url);
         projects.push(pyproject);
     }
@@ -263,15 +268,20 @@ pub fn update_single_project_lockfile(
         echo!("Generating {} lockfile: {}", lock_mode, lockfile.display());
     }
 
-    let features_by_project = collect_workspace_features(lock_options);
-    let applicable_extras = format_project_extras(features_by_project.as_ref(), pyproject)?;
     let mut req_file = NamedTempFile::new()?;
-    writeln!(
-        req_file,
-        "-e {}{}",
-        make_relative_url(&pyproject.root_path(), &pyproject.workspace_path())?,
-        applicable_extras
-    )?;
+
+    // virtual packages are themselves not installed
+    if !pyproject.is_virtual() {
+        let features_by_project = collect_workspace_features(lock_options);
+        let applicable_extras = format_project_extras(features_by_project.as_ref(), pyproject)?;
+        writeln!(
+            req_file,
+            "-e {}{}",
+            make_relative_url(&pyproject.root_path(), &pyproject.workspace_path())?,
+            applicable_extras
+        )?;
+    }
+
     for dep in pyproject.iter_dependencies(DependencyKind::Normal) {
         writeln!(req_file, "{}", dep)?;
     }

rye/src/pyproject.rs

@@ -952,6 +952,16 @@ impl PyProject {
         }
     }
 
+    /// Is this a virtual package (does not build)
+    pub fn is_virtual(&self) -> bool {
+        self.doc
+            .get("tool")
+            .and_then(|x| x.get("rye"))
+            .and_then(|x| x.get("virtual"))
+            .and_then(|x| x.as_bool())
+            .unwrap_or(false)
+    }
+
     /// Should requirements.txt based locking include a find-links reference?
     pub fn lock_with_sources(&self) -> bool {
         match self.workspace {