Skip to content

Commit

Permalink
Replace mentions of src/requirements.txt in documentation
Browse files Browse the repository at this point in the history
Signed-off-by: Juan Luis Cano Rodríguez <juan_luis_cano@mckinsey.com>
  • Loading branch information
astrojuanlu committed May 26, 2023
1 parent 5559e15 commit 4fc7cb6
Show file tree
Hide file tree
Showing 22 changed files with 60 additions and 52 deletions.
2 changes: 1 addition & 1 deletion docs/source/deployment/airflow_astronomer.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ To follow this tutorial, ensure you have the following:
pip install kedro-airflow~=0.4
```

5. Run `pip install -r src/requirements.txt` to install all dependencies.
5. Run `pip install -r requirements.txt` to install all dependencies.

### Deployment process

Expand Down
2 changes: 1 addition & 1 deletion docs/source/deployment/aws_step_functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ The rest of the tutorial will explain each step in the deployment process above

* Create a `conf/aws` directory in your Kedro project
* Put a `catalog.yml` file in this directory with the following content
* Ensure that you have `s3fs>=0.3.0,<0.5` defined in your `src/requirements.txt` so the data can be read from S3.
* Ensure that you have `s3fs>=0.3.0,<0.5` defined in your `requirements.txt` so the data can be read from S3.

<details>
<summary><b>Click to expand</b></summary>
Expand Down
2 changes: 1 addition & 1 deletion docs/source/deployment/distributed.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ For better dependency management, we encourage you to containerise the entire pi
Firstly make sure your [project requirements are up-to-date](../kedro_project_setup/dependencies.md) by running:

```bash
pip-compile --output-file=<project_root>/src/requirements.txt --input-file=<project_root>/src/requirements.txt
pip-compile --output-file=<project_root>/requirements.txt --input-file=<project_root>/requirements.txt
```

We then recommend the [`Kedro-Docker`](https://github.com/kedro-org/kedro-plugins/tree/main/kedro-docker) plugin to streamline the process of building the image. [Instructions for using this are in the plugin's README.md](https://github.com/kedro-org/kedro-plugins/blob/main/README.md).
Expand Down
2 changes: 1 addition & 1 deletion docs/source/deployment/single_machine.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ conda install -c conda-forge kedro
Install the project’s dependencies, by running the following in the project's root directory:

```console
pip install -r src/requirements.txt
pip install -r requirements.txt
```

After having installed your project on the remote server you can run the Kedro project as follows from the root of the project:
Expand Down
8 changes: 4 additions & 4 deletions docs/source/development/commands_reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -292,18 +292,18 @@ _This command will be deprecated from Kedro version 0.19.0._
kedro build-reqs
```

This command runs [`pip-compile`](https://github.com/jazzband/pip-tools#example-usage-for-pip-compile) on the project's `src/requirements.txt` file and will create `src/requirements.lock` with the compiled requirements.
This command runs [`pip-compile`](https://github.com/jazzband/pip-tools#example-usage-for-pip-compile) on the project's `requirements.txt` file and will create `requirements.lock` with the compiled requirements.

`kedro build-reqs` has two optional arguments to specify which file to compile the requirements from and where to save the compiled requirements to. These arguments are `--input-file` and `--output-file` respectively.

`kedro build-reqs` also accepts and passes through CLI options accepted by `pip-compile`. For example, `kedro build-reqs --generate-hashes` will call `pip-compile --output-file=src/requirements.lock --generate-hashes src/requirements.txt`.
`kedro build-reqs` also accepts and passes through CLI options accepted by `pip-compile`. For example, `kedro build-reqs --generate-hashes` will call `pip-compile --output-file=requirements.lock --generate-hashes requirements.txt`.

#### Install all package dependencies

The following runs [`pip`](https://github.com/pypa/pip) to install all package dependencies specified in `src/requirements.txt`:
The following runs [`pip`](https://github.com/pypa/pip) to install all package dependencies specified in `requirements.txt`:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

For further information, see the [documentation on installing project-specific dependencies](../kedro_project_setup/dependencies.md#install-project-specific-dependencies).
Expand Down
6 changes: 3 additions & 3 deletions docs/source/development/linting.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ and check the [cyclomatic complexity](https://www.ibm.com/docs/en/raa/6.1?topic=
type. [You can read more in the `isort` documentation](https://pycqa.github.io/isort/).

### Install the tools
Install `black`, `flake8`, and `isort` by adding the following lines to your project's `src/requirements.txt`
Install `black`, `flake8`, and `isort` by adding the following lines to your project's `requirements.txt`
file:
```text
black # Used for formatting code
Expand All @@ -33,7 +33,7 @@ isort # Used for formatting code (sorting module imports)
To install all the project-specific dependencies, including the linting tools, navigate to the root directory of the
project and run:
```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```
Alternatively, you can individually install the linting tools using the following shell commands:
```bash
Expand Down Expand Up @@ -72,7 +72,7 @@ These hooks are run before committing your code to your repositories to automati
making code reviews easier and less time-consuming.

### Install `pre-commit`
You can install `pre-commit` along with other dependencies by including it in the `src/requirements.txt` file of your
You can install `pre-commit` along with other dependencies by including it in the `requirements.txt` file of your
Kedro project by adding the following line:
```text
pre-commit
Expand Down
13 changes: 8 additions & 5 deletions docs/source/experiment_tracking/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ cd spaceflights
Install the project's dependencies:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

## Set up the session store
Expand Down Expand Up @@ -268,17 +268,20 @@ In this section, we illustrate how to compare Matplotlib plots across experiment

### Update the dependencies

Update the `src/requirements.txt` file in your Kedro project by adding the following dataset to enable Matplotlib for your project:
Update the `pyproject.toml` file in your Kedro project by adding the following dataset to enable Matplotlib for your project:

```text
kedro-datasets[matplotlib.MatplotlibWriter]~=1.1
seaborn~=0.12.1
dependencies = [
# ...
"kedro-datasets[matplotlib.MatplotlibWriter]~=1.1",
"seaborn~=0.12.1",
]
```

And install the requirements with:

```bash
pip install -r src/requirements.txt
pip install --editable .
```

### Add a plotting node
Expand Down
4 changes: 2 additions & 2 deletions docs/source/get_started/new_project.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ There are a few ways to create a new project once you have [set up Kedro](instal

Once you've created a project:

* You need to **navigate to its project folder** and **install its dependencies**: `pip install -r src/requirements.txt`
* You need to **navigate to its project folder** and **install its dependencies**: `pip install -r requirements.txt`
* **To run the project**: `kedro run`
* **To visualise the project**: `kedro viz`

Expand Down Expand Up @@ -79,7 +79,7 @@ kedro new --starter=pandas-iris
However you create a Kedro project, once `kedro new` has completed, the next step is to navigate to the project folder (`cd <project-name>`) and install dependencies with `pip` as follows:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

Now run the project:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ Open your newly-created notebook and create **four new cells** inside it. You wi
1. Before you import and run your Python code, you'll need to install your project's dependencies on the cluster attached to your notebook. Your project has a `requirements.txt` file for this purpose. Add the following code to the first new cell to install the dependencies:

```ipython
%pip install -r "/Workspace/Repos/<databricks_username>/iris-databricks/src/requirements.txt"
%pip install -r "/Workspace/Repos/<databricks_username>/iris-databricks/requirements.txt"
```

2. To run your project in your notebook, you must load the Kedro IPython extension. Add the following code to the second new cell to load the IPython extension:
Expand Down
4 changes: 2 additions & 2 deletions docs/source/integrations/databricks_workspace.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,13 +45,13 @@ kedro new --starter=pyspark-iris

### 2. Install dependencies and run locally

Now, as the project has been successfully created, we should move into the project root directory, install project dependencies, and then start a local test run using [Spark local execution mode](https://stackoverflow.com/a/54064507/3364156), which means that all Spark jobs will be executed in a single JVM locally, rather than in a cluster. `pyspark-iris` Kedro starter used to generate the project already has all necessary configuration for it to work, you just need to have `pyspark` Python package installed, which is done for you by `pip install -r src/requirements.txt` command below.
Now, as the project has been successfully created, we should move into the project root directory, install project dependencies, and then start a local test run using [Spark local execution mode](https://stackoverflow.com/a/54064507/3364156), which means that all Spark jobs will be executed in a single JVM locally, rather than in a cluster. `pyspark-iris` Kedro starter used to generate the project already has all necessary configuration for it to work, you just need to have `pyspark` Python package installed, which is done for you by `pip install -r requirements.txt` command below.

```bash
# change the directory to the project root
cd iris-databricks/
# compile and install the project dependencies, this may take a few minutes
pip install -r src/requirements.txt
pip install -r requirements.txt
# start a local run
kedro run
```
Expand Down
19 changes: 11 additions & 8 deletions docs/source/kedro_project_setup/dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,38 +5,41 @@ Both `pip install kedro` and `conda install -c conda-forge kedro` install the co
When you create a project, you then introduce additional dependencies for the tasks it performs.

## Project-specific dependencies
You can specify a project's exact dependencies in the `src/requirements.txt` file to make it easier for you and others to run your project in the future,
and to avoid version conflicts downstream. This can be achieved with the help of [`pip-tools`](https://pypi.org/project/pip-tools/).
You can specify a project's exact dependencies in the `pyproject.toml` file,
as well as any development dependencies in `requirements.txt`,
to make it easier for you and others to run your project in the future and to avoid version conflicts downstream.
This can be achieved with the help of [`pip-tools`](https://pypi.org/project/pip-tools/).

To install `pip-tools` in your virtual environment, run the following command:
```bash
pip install pip-tools
```

To add or remove dependencies to a project, edit the `src/requirements.txt` file, then run the following:
To add or remove dependencies to a project, edit the `requirements.txt` file, then run the following:

```bash
pip-compile --output-file=<project_root>/src/requirements.txt --input-file=<project_root>/src/requirements.txt
pip-compile --output-file=<project_root>/requirements.txt --input-file=<project_root>/requirements.txt
```

This will [pip compile](https://github.com/jazzband/pip-tools#example-usage-for-pip-compile) the requirements listed in
the `src/requirements.txt` file into a `src/requirements.lock` that specifies a list of pinned project dependencies
the `requirements.txt` file into a `requirements.lock` that specifies a list of pinned project dependencies
(those with a strict version). You can also use this command with additional CLI arguments such as `--generate-hashes`
to use `pip`'s Hash Checking Mode or `--upgrade-package` to update specific packages to the latest or specific versions.
[Check out the `pip-tools` documentation](https://pypi.org/project/pip-tools/) for more information.

```{note}
The `src/requirements.txt` file contains "source" requirements, while `src/requirements.lock` contains the compiled version of those and requires no manual updates.
The `requirements.txt` and `pyproject.toml` files contain "source" requirements, while `requirements.lock` contains the compiled version of those and requires no manual updates.
```

To further update the project requirements, modify the `src/requirements.txt` file (not `src/requirements.lock`) and re-run the `pip-compile` command above.
To further update the project requirements, modify the `requirements.txt` file (not `requirements.lock`) and re-run the `pip-compile` command above.


## Install project-specific dependencies

To install the project-specific dependencies, navigate to the root directory of the project and run:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

## Workflow dependencies
Expand Down
2 changes: 1 addition & 1 deletion docs/source/tutorial/spaceflights_tutorial_faqs.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ documentation on how to install relevant dependencies for kedro_datasets.pandas.
https://kedro.readthedocs.io/en/stable/kedro_project_setup/dependencies.html
```

The Kedro Data Catalog is missing [dependencies needed to parse the data](../kedro_project_setup/dependencies.md#install-dependencies-related-to-the-data-catalog). Check that you have [all the project dependencies to `requirements.txt`](./tutorial_template.md#install-project-dependencies) and then call `pip install -r src/requirements.txt` to install them.
The Kedro Data Catalog is missing [dependencies needed to parse the data](../kedro_project_setup/dependencies.md#install-dependencies-related-to-the-data-catalog). Check that you have [all the project dependencies to `requirements.txt`](./tutorial_template.md#install-project-dependencies) and then call `pip install -r requirements.txt` to install them.

### Pipeline run

Expand Down
6 changes: 3 additions & 3 deletions docs/source/tutorial/tutorial_template.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Set up the spaceflights project

This section shows how to create a new project (with `kedro new` using the [Kedro spaceflights starter](https://github.com/kedro-org/kedro-starters/tree/main/spaceflights)) and install project dependencies (with `pip install -r src/requirements.txt`).
This section shows how to create a new project (with `kedro new` using the [Kedro spaceflights starter](https://github.com/kedro-org/kedro-starters/tree/main/spaceflights)) and install project dependencies (with `pip install -r requirements.txt`).

## Create a new project

Expand Down Expand Up @@ -28,7 +28,7 @@ cd spaceflights

Kedro projects have a `requirements.txt` file to specify their dependencies and enable sharable projects by ensuring consistency across Python packages and versions.

The spaceflights project dependencies are stored in `src/requirements.txt`(you may find that the versions differ slightly depending on the version of Kedro):
The spaceflights project dependencies are stored in `requirements.txt`(you may find that the versions differ slightly depending on the version of Kedro):

```text
# code quality packages
Expand Down Expand Up @@ -63,7 +63,7 @@ scikit-learn~=1.0
To install all the project-specific dependencies, run the following from the project root directory:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

## Optional: logging and configuration
Expand Down
2 changes: 1 addition & 1 deletion docs/source/visualisation/kedro-viz_visualisation.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ When prompted for a project name, you can enter anything, but we will assume `Sp
When your project is ready, navigate to the root directory of the project and install the dependencies for the project, which include Kedro-Viz:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

The next step is optional, but useful to check that all is working. Run the full set of pipelines for the tutorial project:
Expand Down
10 changes: 6 additions & 4 deletions docs/source/visualisation/visualise_charts_with_plotly.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ kedro-datasets[plotly.PlotlyDataSet, plotly.JSONDataSet]~=1.1
Navigate to the root directory of the project in your terminal and install the dependencies for the tutorial project:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

### Configure the Data Catalog
Expand Down Expand Up @@ -177,11 +177,13 @@ You can view Matplotlib charts in Kedro-Viz when you use the [Kedro MatplotLibWr

### Update the dependencies

You must update the `src/requirements.txt` file in the Kedro project by adding the following dataset to enable Matplotlib for the project:
You must update the `pyproject.toml` file in the Kedro project by adding the following dataset to enable Matplotlib for the project:

```bash
kedro-datasets[matplotlib.MatplotlibWriter]~=1.1
seaborn~=0.12.1
dependencies = [
"kedro-datasets[matplotlib.MatplotlibWriter]~=1.1",
"seaborn~=0.12.1",
]
```

### Configure the Data Catalog
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,12 @@ In order to get the best out of the template:

## How to install dependencies

Declare any dependencies in `src/requirements.txt` for `pip` installation and `src/environment.yml` for `conda` installation.
Declare any dependencies in `requirements.txt` for `pip` installation.

To install them, run:

```
pip install -r src/requirements.txt
pip install -r requirements.txt
```

## How to run Kedro
Expand Down Expand Up @@ -52,9 +52,9 @@ To generate or update the dependency requirements for your project:
kedro build-reqs
```

This will `pip-compile` the contents of `src/requirements.txt` into a new file `src/requirements.lock`. You can see the output of the resolution by opening `src/requirements.lock`.
This will `pip-compile` the contents of `requirements.txt` into a new file `requirements.lock`. You can see the output of the resolution by opening `requirements.lock`.

After this, if you'd like to update your project requirements, please update `src/requirements.txt` and re-run `kedro build-reqs`.
After this, if you'd like to update your project requirements, please update `requirements.txt` and re-run `kedro build-reqs`.

[Further information about project dependencies](https://kedro.readthedocs.io/en/stable/kedro_project_setup/dependencies.html#project-specific-dependencies)

Expand Down
2 changes: 1 addition & 1 deletion kedro/framework/cli/micropkg.py
Original file line number Diff line number Diff line change
Expand Up @@ -849,7 +849,7 @@ def _append_package_reqs(
file.write(sep.join(sorted_reqs))

click.secho(
"Use 'kedro build-reqs' to compile and 'pip install -r src/requirements.lock' to install "
"Use 'kedro build-reqs' to compile and 'pip install -r requirements.lock' to install "
"the updated list of requirements."
)

Expand Down
6 changes: 3 additions & 3 deletions kedro/framework/cli/project.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
from kedro.utils import load_obj

NO_DEPENDENCY_MESSAGE = """{module} is not installed. Please make sure {module} is in
{src}/requirements.txt and run 'pip install -r src/requirements.txt'."""
{src}/requirements.txt and run 'pip install -r requirements.txt'."""
LINT_CHECK_ONLY_HELP = """Check the files for style guide violations, unsorted /
unformatted imports, and unblackened Python code without modifying the files."""
OPEN_ARG_HELP = """Open the documentation in your default browser after building."""
Expand Down Expand Up @@ -242,8 +242,8 @@ def build_docs(metadata: ProjectMetadata, open_docs):
def build_reqs(
metadata: ProjectMetadata, input_file, output_file, args, **kwargs
): # pylint: disable=unused-argument
"""Run `pip-compile` on src/requirements.txt or the user defined input file and save
the compiled requirements to src/requirements.lock or the user defined output file.
"""Run `pip-compile` on requirements.txt or the user defined input file and save
the compiled requirements to requirements.lock or the user defined output file.
(DEPRECATED)
"""
deprecation_message = (
Expand Down
2 changes: 1 addition & 1 deletion kedro/framework/cli/starters.py
Original file line number Diff line number Diff line change
Expand Up @@ -370,7 +370,7 @@ def _create_project(template_path: str, cookiecutter_args: dict[str, Any]):
)
click.secho(
"\nA best-practice setup includes initialising git and creating "
"a virtual environment before running 'pip install -r src/requirements.txt' to install "
"a virtual environment before running 'pip install -r requirements.txt' to install "
"project-specific dependencies. Refer to the Kedro documentation: "
"https://kedro.readthedocs.io/"
)
Expand Down
2 changes: 1 addition & 1 deletion kedro/framework/cli/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -354,7 +354,7 @@ def _check_module_importable(module_name: str) -> None:
except ImportError as exc:
raise KedroCliError(
f"Module '{module_name}' not found. Make sure to install required project "
f"dependencies by running the 'pip install -r src/requirements.txt' command first."
f"dependencies by running the 'pip install -r requirements.txt' command first."
) from exc


Expand Down
Loading

0 comments on commit 4fc7cb6

Please sign in to comment.