diff --git a/pages/custom-query-modules.mdx b/pages/custom-query-modules.mdx index ff41688c1..a921d8225 100644 --- a/pages/custom-query-modules.mdx +++ b/pages/custom-query-modules.mdx @@ -4,22 +4,26 @@ description: Expand the capabilities of Memgraph through custom query modules. E --- import { Callout } from 'nextra/components' +import { Steps } from 'nextra/components' +import { Tabs, Tab } from 'nextra/components' # Custom query modules -Memgraph supports extending the query language with user-written procedures in -**C**, **C++**, **Python**, and **Rust**. These procedures are grouped into -modules - **query modules** files (either `*.so` or `*.py` files). +Many [graph algorithms and uitility +procedures](/advanced-algorithms/available-algorithms) have already been +developed and are available as part of the MAGE library you can add to your +Memgraph installation. The library is already included if you are using Memgraph +Platform or Memgraph MAGE Docker images to run Memgraph. -Some query modules are built-in, and others, like those that can help you solve -complex graph issues, are available as part of the MAGE library you can add to -your Memgraph installation. The library is already included if you are using -Memgraph Platform or Memgraph MAGE Docker images to run Memgraph. +But, you can also implement custom query modules. Memgraph supports extending +the query language with user-written procedures in **C**, **C++**, **Python**, +and **Rust**. These procedures are grouped into modules - **query module** files +(either `*.so` or `*.py` files). -You can also implement custom query modules. Every single Memgraph installation -comes with the `example.so` and `py_example.py` query modules located in the -`/usr/lib/memgraph/query_modules` directory. They were provided as examples of -query modules for you to examine and learn from. +Every single Memgraph installation comes with the `example.so` and +`py_example.py` query modules located in the `/usr/lib/memgraph/query_modules` +directory. They were provided as examples of query modules for you to examine +and learn from. Each query module file corresponds to one query module, and file names are mapped as query module names. For example, `example.so` will be mapped as @@ -28,376 +32,435 @@ each module file has a procedure called `procedure` defined, those procedures would be mapped in the Cypher query language as `example.procedure()` and `py_example.procedure()` respectively. -Regardless of where they come from and who wrote them, all modules need to be -loaded into Memgraph so that they can be called while querying the database. -They are either loaded automatically when Memgraph starts or manually if they -were added while Memgraph was already running. +You can develop custom procedures from Memgraph itself. -You can also inspect and develop query modules in Memgraph Lab (v2.0 and newer). -Just navigate to **Query Modules**. +## Develop custom query modules + +If you want to develop Python procedures you can do so from [Memgraph +Lab](/data-visualization), the visual user interface. The files will be saved at +`/var/lib/memgraph/internal_modules`. Just navigate to **Query Modules** and +select *+ New Module*. -Screenshot of Query Modules from Memgraph Lab ![](/pages/custom-query-modules/memgraph_lab_query_modules.png) +If you don't won't to develop from Memgraph Lab, you can: +- install MAGE development environment from Docker Hub +- build MAGE using the `docker build` command +- build MAGE from source -Once you start Memgraph, it will attempt to load query modules from all *.so and -*.py files from the default directories. MAGE modules are located at -`/usr/lib/memgraph/query_modules` and custom modules developed via Memgraph Lab at -`/var/lib/memgraph/internal_modules`. +Then, select a language you want to develop in. -Memgraph provides public APIs for writing custom query modules in Python, C and -C++. +### Install MAGE and import query modules -## Python API + -Python API is defined in the `mgp` module that can be found in the Memgraph -installation directory `/usr/lib/memgraph/python_support`. In essence, it is a -wrapper around the C API. If you wish to write your own query modules using the -Python API, you need to have Python version `3.5.0` or above installed. + -For more information, check the [Python API reference -guide](/custom-query-modules/python/python-api).
-We also made [an example module](/custom-query-modules/python/python-example) to -help you start developing your own modules. + The following steps will guide you through installing MAGE development + environment from Docker Hub and importing develooped query modules. + + -You can develop query modules in Python from Memgraph Lab (v2.0 and newer). Just -navigate to **Query Modules** and click on **New Module** to start. + {

Download the MAGE image

} + + Run the following command to get the Memgraph and MAGE development Docker image: -![](/pages/custom-query-modules/memgraph_lab_query_modules.png) + ```shell + docker run -p 7687:7687 memgraph/memgraph-mage:-dev + ``` -Custom modules developed via Memgraph Lab are located at -`/var/lib/memgraph/internal_modules`. + By running this command, you will get an image with the following tools + installed: `Python3`, `Rust`, `Clang`, `Make`, and `CMake`. - -If you need an additional Python library not included with Memgraph, check out -[the guide on how to install -it](/custom-query-modules/manage-query-modules). - + {

Develop a query

} -### Mock Python API + Develop query modules in [Python](/custom-query-modules/python), + [C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or + [Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). -The [mock Python query module API](/custom-query-modules/python/mock-python-api) enables you to -develop and test query modules for Memgraph without having to run a Memgraph -instance by simulating its behavior. As the mock API is compatible with the -[Python API](/custom-query-modules/python/python-api), -you can add modules developed with it to Memgraph as-is, without modifying the -code. + {

Start the MAGE container

} -For more information and examples, check the -[mock Python API reference guide](/custom-query-modules/python/mock-python-api). + Use the following command to start the MAGE container: -## C API + ```shell + docker run --rm -p 7687:7687 --name mage memgraph-mage:version-dev + ``` -C API modules need to be compiled to a shared library so that they can be loaded -when Memgraph starts. This means that you can write the procedures in any -programming language that can work with C and be compiled to the ELF shared -library format (`.so`). `mg_procedure.h` that can be found in Memgraph -installation directory `/usr/include/memgraph` contains declarations of all -functions that can be used to implement a query module procedure. To compile the -module, you will have to pass the appropriate flags to the compiler, for -example, `clang`: + Be sure to replace the version with the specific version, for example: -```plaintext -clang -Wall -shared -fPIC -I /usr/include/memgraph example.c -o example.so -``` + ```shell + docker run --rm -p 7687:7687 --name mage memgraph-mage:1.4-dev + ``` -For more information, check the [C API reference -guide](/custom-query-modules/c/c-api).
-We also made [an example module](/custom-query-modules/c/c-example) to help you -start developing your own modules. + {

Copy the files to the container

} -## C++ API + Copy the files to the container named `mage`: -C++ API modules, just like C API modules, need to be compiled to a shared -library so that they can be loaded when Memgraph starts. This is done much in -the same way as with C API modules. + ```shell + docker cp . mage:/mage/ + ``` -For more information, check the [C++ API reference -guide](/custom-query-modules/cpp/cpp-api).
-We also made [an example -module](/custom-query-modules/cpp/cpp-example) -to help you start developing your own modules. + {

Enter the container

} - -If you need an additional Python library not included with Memgraph, check out -[the guide on how to install -it](/custom-query-modules/manage-query-modules). - + Position yourself inside the container as `root`: -## Install MAGE + ```shell + docker exec -u root -it mage /bin/bash + ``` + -You can also download a MAGE image equipped for development inside of Docker -containers: + If you performed the build locally, make sure to delete the `cpp/build` directory because you might be dealing with different architectures or problems with `CMakeCache.txt`. -```shell -docker run -p 7687:7687 memgraph/memgraph-mage:1.1-dev -``` + To delete it, run: -By running this command, you will get an image with the following tools -installed: Python3, Rust, Clang, Make, and CMake. This way, you can copy files -to the container, build them inside and import query modules in Memgraph. + ```shell + rm -rf cpp/build + ``` -If you want to develop your own query modules, be sure to check the [Development -process for MAGE with -Docker](https://github.com/memgraph/mage#developing-mage-with-docker). + + {

Build MAGE

} ----- + Build MAGE with the option to copy executables from `mage/dist` to `/usr/lib/memgraph/query_modules`: -## This is something taken from Install MAGE that seems it's used for building custom query_modules + ```shell + python3 setup build -p /usr/lib/memgraph/query_modules/ + ``` -## This is when building MAGE from source with Linux - + {

Exit the container

} -Be sure you cloned the `mage` GitHub repository using the `--recurse-submodules` flag since it has incorporated Memgraph inside: + Everything should be ready to exit the container and load the query modules: -```shell -git clone --recurse-submodules https://github.com/memgraph/mage.git -``` + ```shell + exit + ``` -if you didn't, you can run following command to update submodules: + -```shell -git submodule update --init --recursive -``` - + + + Install MAGE with Docker Build. -**2.** Start Memgraph and enjoy **MAGE**! + Create a Docker image directly from the [MAGE GitHub + repository](https://github.com/memgraph/mage), instead of pulling it from the + Docker Hub. - -Query modules are loaded into Memgraph on startup, so if your instance -was already running you will need to execute the following query inside one of -[querying platforms](/) to -load them: + -``` -CALL mg.load_all(); -``` + {

Download image or clone the repo

} -If your changes are not loaded, make sure to restart the instance by running -`systemctl stop memgraph` and `systemctl start memgraph`. + Download a [specific release](https://github.com/memgraph/mage/releases) from + the MAGE repository or clone the [repository](https://github.com/memgraph/mage) for the latest + version. -If you want to find out more about loading query modules, visit [this -guide](/custom-query-modules/manage-query-modules). + ```shell + git clone --recurse-submodules https://github.com/memgraph/mage.git && cd mage + ``` + {

Build the MAGE tagged Docker image

} - + Run the the following command: -### Advanced configuration + ```shell + docker build -t memgraph-mage . + ``` -#### 1. Set a different `query_modules` directory + Verify that the build is successful by starting the built image: -The `setup` script can set your local `mage/dist` directory or **any** other -directory as the **default** one in the Memgraph configuration file (flag -`--query-modules-directory` defined in `/etc/memgraph/memgraph.conf`). There are -a few options: + ```shell + docker run --rm -p 7687:7687 --name mage memgraph-mage + ``` -**1.** Set `` as the **default** one: + {

Develop modules

} -``` -python3 setup modules_storage -p -``` + Develop query modules in [Python](/custom-query-modules/python), + [C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or + [Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). -This way Memgraph will be looking for query modules inside ``. + {

Create the `dev` image

} + + To create the `dev` MAGE image, run the following command: - + ```shell + docker build --target dev -t memgraph-mage:dev . + ``` -Don't forget to copy the aforementioned files from `mage/dist` to -``. + {

Start the container

} + + Start the container with the following command: - + ```shell + docker run --rm -p 7687:7687 --name mage memgraph-mage:dev + ``` -**2.** Set `/mage/dist` as the **default** one: + {

Copy the files to the container

} -``` -python3 setup modules_storage -``` + Copy the files to the `mage` container: -If the **default** directory is `mage/dist` then you don't need to copy `*.so` -and `*.py` files from the `mage/dist` directory -to`/usr/lib/memgraph/query_modules` every time you run `build`. + ```shell + docker cp . mage:/mage/ + ``` -## Docker build + {

Enter the container

} -This way, you will create a Docker image directly from the [MAGE Github -repository](https://github.com/memgraph/mage) and won't have to pull it from -Docker Hub. You can: + Position yourself inside the container as `root`: -- download a [specific release](https://github.com/memgraph/mage/releases) from - the MAGE repository or -- clone the [repository](https://github.com/memgraph/mage) for the latest - version. + ```shell + docker exec -u root -it mage /bin/bash + ``` + -If you downloaded a specific release, skip the first step. + If you performed the build locally, make sure to delete the `cpp/build` directory because you might be dealing with different architectures or problems with `CMakeCache.txt`. -## Installing MAGE + To delete it, run: -**1.** Download the MAGE source code from -**[GitHub](https://github.com/memgraph/mage)**: + ```shell + rm -rf cpp/build + ``` -```shell -git clone --recurse-submodules https://github.com/memgraph/mage.git && cd mage -``` + -**2.** Build the **MAGE** tagged Docker image with the following command: + {

Build MAGE

} -```shell -docker build -t memgraph-mage . -``` + Build MAGE with the option to copy executables from `mage/dist` to `/usr/lib/memgraph/query_modules`: -**3.** Start Memgraph-MAGE with the following command: + ```shell + python3 setup build -p /usr/lib/memgraph/query_modules/ + ``` -```shell -docker run --rm -p 7687:7687 --name mage memgraph-mage -``` + {

Exit the container

} - + Everything should be ready to exit the container and load the query modules: -Now you can query Memgraph with any of the querying platforms like [Memgraph -Lab](https://memgraph.com/product/lab) or -[mgconsole](https://github.com/memgraph/mgconsole). + ```shell + exit + ``` -If you made any changes while the **MAGE** Docker container was running, you -would need to stop the Docker container and rebuild the whole image. If you -don't want to repeat these steps each time, be sure to check the [Development -process for MAGE with -Docker](https://github.com/memgraph/mage#developing-mage-with-docker). + + + + + Follow the steps if you want to use the MAGE library with [installed Linux based + Memgraph package](/getting-started/install-memgraph) and create custom query modules. + + + + {

Install dependencies

} + + To build from source, you will need: + - Python3 + - Make + - CMake + - Clang + - UUID + - [Rust and Cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html) + + {

Set up the machine

} + + Run the following commands: + + ```bash + sudo apt-get update && apt-get install -y \ + libcurl4 `memgraph` \ + libpython${PY_VERSION} `memgraph` \ + libssl-dev `memgraph` \ + openssl `memgraph` \ + build-essential `mage-memgraph` \ + cmake `mage-memgraph` \ + curl `mage-memgraph` \ + g++ `mage-memgraph` \ + python3 `mage-memgraph` \ + python3-pip `mage-memgraph` \ + python3-setuptools `mage-memgraph` \ + python3-dev `mage-memgraph` \ + clang `mage-memgraph` \ + git `mage-memgraph` \ + --no-install-recommends \ + && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* + ``` - + {

Download the MAGE source code

} -## Developing MAGE with Docker + Clone the [MAGE source code](https://github.com/memgraph/mage) from GitHub: -When developing your query module, you need to load it inside Memgraph running -inside the Docker container. You can do that by [rebuilding the whole MAGE -image](#1-rebuild-the-whole-mage-image) or by [building it inside the Docker -container](#2-build-inside-the-docker-container). + ``` + git clone --recurse-submodules https://github.com/memgraph/mage.git + ``` -### 1. Rebuild the whole MAGE image + {

Run the `setup` script

} -This command will trigger the rebuild of the whole Docker image. Make sure that -you have added Python requirements inside `python/requirements.txt` file. + Run the following command: -**1.** Firstly, do the build of the **MAGE** image: + ```shell + python3 setup build -p /usr/lib/memgraph/query_modules + ``` -``` -docker build -t memgraph-mage . -``` + The script will generate a `dist` directory with all the necessary files. -**2.** Now, start `memgraph-mage` image with the following command and enjoy -**your** own **MAGE**: + It will also copy the contents of the newly created `dist` directory to + `/usr/lib/memgraph/query_modules`. Memgraph loads algorithms and modules from + this directory. -``` -docker run --rm -p 7687:7687 --name mage memgraph-mage -``` + If something isn't installed properly, the `setup` script will stop the + installation process. If you have any questions, contact us on + **[Discord](https://discord.gg/memgraph).** -### 2. Build inside the Docker container + {
Set a different `query_modules` directory
} -You can build a **MAGE** Docker image equipped for development. `Rust`, `Clang`, -`Python3-pip`, and everything else necessary for development will still be -inside the running container. This means that you can copy the **MAGE** -repository to the container and do the build inside the `mage` container. There -is no need to do the whole Docker image build again. + The `setup` script can set your local `mage/dist` directory or any other + directory as the default one by changing the value of the + `--query-modules-directory` flag in the `/etc/memgraph/memgraph.conf`, + Memgraph's configuration file. -**1.** To create `dev` **MAGE** image, run the following command: + By setting the `/mage/dist` as the default directory you don't need to copy + `*.so` and `*.py` files from the `mage/dist` directory + to`/usr/lib/memgraph/query_modules` every time you run `build`: -``` -docker build --target dev -t memgraph-mage:dev . -``` + ``` + python3 setup modules_storage + ``` -**2.** Then run the image with the following command: + By setting `` as the default one, Memgraph will be looking for + query modules inside ``, instead of `/usr/lib/memgraph/query_modules`: -``` -docker run --rm -p 7687:7687 --name mage memgraph-mage:dev -``` + ``` + python3 setup modules_storage -p + ``` -**3.** Next, copy the files inside the container and do the build: + Don't forget to copy the aforementioned files from `mage/dist` to + ``. -**a)** First, you need to copy the files to the container named `mage` + {

Develop modules

} -``` -docker cp . mage:/mage/ -``` + Develop query modules in [Python](/custom-query-modules/python), + [C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or + [Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). -**b)** Then, you need to position yourself inside the container as root: + {

Start Memgraph

} + + Make sure your Memgraph instance is running: -``` -docker exec -u root -it mage /bin/bash -``` + ``` + sudo systemctl status memgraph.service + ``` - + {

Copy the query module

} + + Copy your developed query module to `/usr/lib/memgraph/query_modules` or your directory if you changed the default location of query modules: -Note: If you have done the build locally, make sure to delete the directory -`cpp/build` because you might be dealing with different `architectures` or -problems with `CMakeCache.txt`. To delete it, run: + ```shell + python3 setup build -p /usr/lib/memgraph/query_modules + ``` -`rm -rf cpp/build` + - + + + -**c)** After that, run build and copy `mage/dist` to -`/usr/lib/memgraph/query_modules`: + {

Download MAGE source code

} + Download the MAGE source code from GitHub: -``` -python3 setup build -p /usr/lib/memgraph/query_modules/ -``` + ```shell + git clone https://github.com/memgraph/mage.git && cd mage + ``` -**d)** Everything should be ready, and you can run the following command to exit -the container: + {

Build the image

} + + Build the *MAGE × cuGraph*-tagged Docker image: + + ```shell + docker build -f Dockerfile.cugraph -t memgraph-mage . + ``` -``` -exit -``` + {

Run the image

} - + Start Memgraph-MAGE with the following command: -Note that query modules are loaded into Memgraph on startup, so if your instance -was already running, you would need to execute the following query inside one of -the [querying platforms](/) -to load them: + ```shell + docker run --rm --gpus all -p 7687:7687 -p 7444:7444 --name mage memgraph-mage + ``` + -`CALL mg.load_all();` + If you made any changes while the Docker container was running, you need to stop + the container and rebuild the image. - + + + -## Memgraph X cuGraph - +## Querying -The development image with cuGraph support is not available yet. If you want to -develop cuGraph-powered query modules in Docker, do not hesitate to [contact -us](https://memgraph.com/community) about it. +Query modules are loaded into Memgraph on startup, so if your instance was +already running, you need to execute the following query inside one of the +querying platforms ([mgonsole](/getting-started/cli), [Memgraph +Lab](/data-visualization) or a [client library](/client-libraries)) to load +them: - +```cypher +CALL mg.load_all(); +``` -### Building MAGE with NVIDIA cuGraph locally with Docker +Run a query and test your module. All procedures are called using the `CALL` +clause: -1. Download the MAGE source code from - [GitHub](https://github.com/memgraph/mage): +```cypher +MERGE (start:Node {id: 0})-[:RELATION]->(:Node {id: 1})-[:RELATION]->(:Node {id: 2}) +CALL random_walk.get(start, 2) YIELD path +RETURN path +``` - ```shell - git clone https://github.com/memgraph/mage.git && cd mage - ``` +Since every query module is run as one transaction in Memgraph, the user can +stop the query module by [terminating the corresponding +transaction](/memgraph/reference-guide/transactions), but only if crucial parts +of the code, such as while and until loops, or similar points where the +procedure might become costly, are preceded with CheckMustAbort() function. The +user can then find out the transaction ID using `SHOW TRANSACTIONS` command and +terminate it using the `TERMINATE TRANSACTIONS ` command. -2. Build the **MAGE × cuGraph**-tagged Docker image: +## Testing - ```shell - docker build -f Dockerfile.cugraph -t memgraph-mage . - ``` +Test decoupled parts of your code that don't depend on Memgraph like you would +in any other setting. E2e (end to end) tests, on the other hand, depend on +internal Memgraph data structures, like nodes and edges. After running Memgraph, +we need to prepare the testing environment on the host machine. Position +yourself in the mage directory you cloned from GitHub. The expected folder +structure for each module is the following: -3. Start Memgraph-MAGE with the following command: - ```shell - docker run --rm --gpus all -p 7687:7687 -p 7444:7444 --name mage memgraph-mage - ``` +```plaintext +mage +└── e2e + └── random_walk_test + └── test_base + ├── input.cyp + └── test.yml +``` + +`input.cyp` represents a Cypher script for entering the data into the database. +To simplify this tutorial, we'll leave the database empty. `test.yml` specifies +which test query should be run by the database and what should be the result or +exception. Create the files following the aforementioned directory structure. + +### input.cyp - +```cypher +MATCH (n) DETACH DELETE n; +``` + +### test.yml + +```shell +query: > + MATCH (start:Node {id: 0}) + CALL random_walk.get(start, 2) YIELD path + RETURN path -You can now query Memgraph from querying platforms such as [Memgraph -Lab](https://memgraph.com/product/lab) or -[mgconsole](https://github.com/memgraph/mgconsole). +output: [] +``` -If you made any changes while the Docker container was running, you need to stop -the container and rebuild the image. For a workaround, check [Development -process for MAGE with -Docker](/advanced-algorithms/install-mage). +Lastly, run the e2e tests with python: - \ No newline at end of file +```shell +python test_e2e +``` \ No newline at end of file diff --git a/pages/custom-query-modules/c.mdx b/pages/custom-query-modules/c.mdx new file mode 100644 index 000000000..edc4ace4b --- /dev/null +++ b/pages/custom-query-modules/c.mdx @@ -0,0 +1,19 @@ +# How to create a query module in C++ + +C API modules need to be compiled to a shared library so that they can be loaded +when Memgraph starts. This means that you can write the procedures in any +programming language that can work with C and be compiled to the ELF shared +library format (`.so`). `mg_procedure.h` that can be found in Memgraph +installation directory `/usr/include/memgraph` contains declarations of all +functions that can be used to implement a query module procedure. To compile the +module, you will have to pass the appropriate flags to the compiler, for +example, `clang`: + +```plaintext +clang -Wall -shared -fPIC -I /usr/include/memgraph example.c -o example.so +``` + +For more information, check the [C API reference +guide](/custom-query-modules/c/c-api).
+We also made [an example module](/custom-query-modules/c/c-example) to help you +start developing your own modules. \ No newline at end of file diff --git a/pages/custom-query-modules/contributing.mdx b/pages/custom-query-modules/contributing.mdx index 67f0e0d24..596accfc7 100644 --- a/pages/custom-query-modules/contributing.mdx +++ b/pages/custom-query-modules/contributing.mdx @@ -21,205 +21,16 @@ and your contribution: - [**Memgraph**](https://github.com/memgraph/memgraph) - [**MAGE**](https://github.com/memgraph/mage) +Feel free to create an issue or open a pull request on our Github +repo to speed up the development. +Also, don't forget to throw us a star on GitHub. + +Your feedback is always welcome and valuable to us. Please don't hesitate to +post on [Discord](https://www.discord.gg/memgraph). + ## Code of Conduct Everyone participating in this project is governed by the [Code of Conduct](https://github.com/memgraph/mage/blob/main/CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable -behavior to tech@memgraph.com. - -## Prerequisites - -- You have developed a query module by yourself and/or followed our tutorial for - [Python](/custom-query-modules/cpp) or - [C++](/custom-query-modules/cpp) - - - -The following steps depend on how you installed Memgraph and MAGE as we need -to import the modules. - - - -## Importing query modules into Memgraph - - - - - -**1.** Start the MAGE container with: - -```shell -docker run --rm -p 7687:7687 --name mage memgraph-mage:version-dev -``` - -Be sure to replace the `version` with the specific version, for example: - -```shell -docker run --rm -p 7687:7687 --name mage memgraph-mage:1.4-dev -``` - -**2.** Copy your local MAGE directory inside the container in order for Memgraph -to import the query modules: - -**a)** First, you need to copy the files to the container named `mage`: - -```shell -docker cp . mage:/mage/ -``` - -**b)** Then, you need to position yourself inside the container as root: - -```shell -docker exec -u root -it mage /bin/bash -``` - -> Note: If you performed the build locally, make sure to delete the `cpp/build` -> directory because you might be dealing with different architectures or -> problems with `CMakeCache.txt`. To delete it, run: -> -> `rm -rf cpp/build` - -**c)** After that, build MAGE with the option to copy executables from -`mage/dist` to `/usr/lib/memgraph/query_modules`: - -```shell -python3 setup build -p /usr/lib/memgraph/query_modules/ -``` - -**d)** Everything should be ready to exit the container and load the query -modules: - -``` -exit -``` - - - - -**1.** To create the `dev` **MAGE** image, run the following command: - -```shell -docker build --target dev -t memgraph-mage:dev . -``` - -**2.** Start the container with the following command: - -```shell -docker run --rm -p 7687:7687 --name mage memgraph-mage:dev -``` - - - -If you make any changes to the module, you can stop the container and do a -rebuild. Additionally, if you don't want to rebuild everything, you can: -1. Copy the changes to the container. -2. Perform a build inside the container. -3. Copy the executables to the `/usr/lib/memgraph/query_modules/` directory, - where Memgraph is looking for query modules. - -The process is the same as described in step **2** of the tab `Docker Hub`. - - - - - - -**1.** Make sure your Memgraph instance is running: - -``` -sudo systemctl status memgraph.service -``` - -**2.** Now, we need to copy our developed query module `random_walk.py` to -`/usr/lib/memgraph/query_modules`: - -```shell -python3 setup build -p /usr/lib/memgraph/query_modules -``` - - - - - -## Querying - -> Note that query modules are loaded into Memgraph on startup, so if your -> instance was already running, you would need to execute the following query -> inside one of the [querying -> platforms](/) to load -> them: - -```cypher -CALL mg.load_all(); -``` - -Lastly, run a query and test your module: - -```cypher -MERGE (start:Node {id: 0})-[:RELATION]->(:Node {id: 1})-[:RELATION]->(:Node {id: 2}) -CALL random_walk.get(start, 2) YIELD path -RETURN path -``` - -Since every query module is run as one transaction in Memgraph, the user can stop -the query module by [terminating the corresponding transaction](/fundamentals/transactions). The user first needs -to find out the transaction ID using `SHOW TRANSACTIONS` command and then terminate it -using the `TERMINATE TRANSACTIONS ` command. - -## Testing - -Test decoupled parts of your code that don't depend on Memgraph like you would -in any other setting. E2e (end to end) tests, on the other hand, depend on -internal Memgraph data structures, like nodes and edges. After running Memgraph, -we need to prepare the testing environment on the host machine. Position -yourself in the mage directory you cloned from GitHub. The expected folder -structure for each module is the following: - -```plaintext -mage -└── e2e - └── random_walk_test - └── test_base - ├── input.cyp - └── test.yml -``` - -`input.cyp` represents a Cypher script for entering the data into the database. -To simplify this tutorial, we'll leave the database empty. `test.yml` specifies -which test query should be run by the database and what should be the result or -exception. Create the files following the aforementioned directory structure. - -### input.cyp - -```cypher -MATCH (n) DETACH DELETE n; -``` - -### test.yml - -```shell -query: > - MATCH (start:Node {id: 0}) - CALL random_walk.get(start, 2) YIELD path - RETURN path - -output: [] -``` - -Lastly, run the e2e tests with python: - -```shell -python test_e2e -``` - -## Next steps - -Feel free to create an issue or open a pull request on our [Github -repo](https://github.com/memgraph/mage) to speed up the development.
-Also, don't forget to throw us a star on GitHub. :star: - - -## Feedback -Your feedback is always welcome and valuable to us. Please don't hesitate to -post on our [Discord](https://www.discord.gg/memgraph). +behavior to tech@memgraph.com. \ No newline at end of file diff --git a/pages/custom-query-modules/cpp.mdx b/pages/custom-query-modules/cpp.mdx index 389f79370..d375b8f92 100644 --- a/pages/custom-query-modules/cpp.mdx +++ b/pages/custom-query-modules/cpp.mdx @@ -7,31 +7,34 @@ import { Callout } from 'nextra/components' # How to create a query module in C++ +## C++ API + Query modules can be implemented using the [C++ -API](/custom-query-modules/cpp/cpp-api) -provided by Memgraph with automatic memory management. -In this tutorial, we will learn how to develop a query -module in C++ on the example of the **random walk algorithm**. +API](/memgraph/reference-guide/query-modules/api/cpp-api) provided by Memgraph +with automatic memory management. C++ API modules, just like C API modules, need +to be compiled to a shared library so that they can be loaded when Memgraph +starts. This is done much in the same way as with C API modules. + +For more information, check the [C++ API reference +guide](/custom-query-modules/cpp/cpp-api).
+We also made [an example module](/custom-query-modules/cpp/cpp-example) to help +you start developing your own modules. + +## Example + +In this tutorial, we will learn how to develop a query module in C++ on the +example of the **random walk algorithm**. ## Prerequisites There are three options for installing and working with Memgraph MAGE: -1. **Pulling the `memgraph/memgraph-mage` image**: check the `Docker Hub` - [installation guide](/advanced-algorithms/install-mage). -2. **Building a Docker image from the MAGE repository**: check the `Docker - build` [installation guide](/advanced-algorithms/install-mage). -3. **Building MAGE from source**: check the `Build from source on Linux` - [installation guide](/advanced-algorithms/install-mage). +- [install MAGE development environment from Docker Hub](/custom-query-modules#install-mage-with-docker-hub) +- [build MAGE using the `docker build` command](/custom-query-modules#install-mage-with-docker-build) +- [build MAGE from source](/custom-query-modules#install-mage-from-source) ## Developing a module - - -These instructions are the same for every MAGE installation option: _Docker -Hub_, _Docker build_ and _Build from source on Linux_. - - Position yourself in the **MAGE repository** you cloned earlier. Once you are there, enter the `cpp` subdirectory and create a new directory called diff --git a/pages/custom-query-modules/manage-query-modules.mdx b/pages/custom-query-modules/manage-query-modules.mdx index 7d80f02b3..bd122bbb7 100644 --- a/pages/custom-query-modules/manage-query-modules.mdx +++ b/pages/custom-query-modules/manage-query-modules.mdx @@ -7,76 +7,6 @@ import { Callout } from 'nextra/components' # Manage query modules -The following page describes how query modules are loaded into Memgraph and -called within a Cypher query. - -## Loading query modules - -Once you start Memgraph, it will attempt to load query modules from all `*.so` -and `*.py` files from the default (`/usr/lib/memgraph/query_modules` and -`/var/lib/memgraph/internal_modules`) directories. - -MAGE modules are located at -`/usr/lib/memgraph/query_modules` and custom modules developed via Memgraph Lab at -`/var/lib/memgraph/internal_modules`. - -Memgraph can load query modules from additional directories, if their path is -added to the `--query-modules-directory` flag in the main configuration file -(`/etc/memgraph/memgraph.conf`) or supplied as a command-line parameter (e.g. -when using Docker). - -If you are supplying the additional directory as a parameter, do not forget to -include the path to `/usr/lib/memgraph/query_modules`, otherwise queries from -that directory will not be loaded when Memgraph starts. - - - -When working with Docker and `memgraph-platform` image, you should pass -configuration flags inside of environment variables, for example: - -```terminal -docker run -p 7687:7687 -p 7444:7444 -p 3000:3000 -e MEMGRAPH="--query-modules-directory=/usr/lib/memgraph/query_modules,/usr/lib/memgraph/my_modules" memgraph/memgraph-platform` -``` - -If you are working with `memgraph` or `memgraph-mage` images you should pass -configuration options like this: - -```terminal -docker run -p 7687:7687 -p 7444:7444 memgraph/memgraph --query-modules-directory=/usr/lib/memgraph/query_modules,/usr/lib/memgraph/my_modules -``` - - - -If a certain query module was added while Memgraph was already running, you need -to load it manually using the `mg.load("module_name")` procedure within a query: - -```cypher -CALL mg.load("py_example"); -``` - -If there is no response (no error message), the load was successful. - -If you want to reload all existing modules and load any newly added ones, use -`mg.load_all()`: - -```cypher -CALL mg.load_all(); -``` - -If there is no response (no error message), the load was successful. - -You can check if the query module has been loaded by using the `mg.procedures()` -procedure within a query: - -```cypher -CALL mg.procedures() YIELD *; -``` - -Built-in utility query module (`mg`) contains procedures that enable you to -manage query modules files. - - -### This is from advanced algorithms, check if there is anything new Here ## Load procedures Once you start Memgraph, it will attempt to load query modules from all `*.so` @@ -147,7 +77,10 @@ syntax: CALL module.procedure([optional parameter], arg1, "string_argument", ...) YIELD res1, res2, ...; ``` -## General procedures +The optional parameter is used if you want to run the procedure on a +[subgraph](/advanced-algorithms/run-algorithms#run-procedures-on-subgraph). + +## General `mg` procedures Here is the list of procedures from the `mg` query module that can be used with all other query module files and their signatures: @@ -230,7 +163,7 @@ CALL mg.load("module1"); ``` Memgraph will reload `module1.py`, all its imported Python packages and it will conclude that there is a subdirectory `module1` which contains Python utility files for `module1.py` and it will reload them too. Note, that if `module1` directory contains some subdirectories, they will also get reloaded. By reloading, it is assumed clearing from the `sys` cache and deleting compiled code from the `__pycache__`. The repository which contains subdirectories can be organized in a different way too, so e.g. `module1/` and `module2/` folders can be placed directly in the `python/` folder. -## Procedures for `.py` query modules +## `mg` procedures for `.py` query modules Memgraph includes several built-in procedures for editing and inspecting Python module files. @@ -368,7 +301,13 @@ syntax: ```cypher CALL module.procedure([optional parameter], arg1, "string_argument", ...) YIELD res1, res2, ...; ``` -Every procedure has a first optional parameter and the rest of them depend on the procedure you are trying to call. The optional parameter must be result of the aggregation function [`project()`](/querying/functions#aggregation-functions). If such a parameter is provided, **all** operations will be executed on a projected graph. Otherwise, you will work on the whole graph stored inside Memgraph. + +Every procedure has a first optional parameter and the rest of them depend on +the procedure you are trying to call. The optional parameter must be result of +the aggregation function +[`project()`](/cypher-manual/functions#aggregation-functions). If such a +parameter is provided, **all** operations will be executed on a projected graph. +Otherwise, you will work on the whole graph stored inside Memgraph. Each procedure returns zero or more records, where each record contains named fields. The `YIELD` clause is used to select fields you are interested in or all diff --git a/pages/custom-query-modules/python.mdx b/pages/custom-query-modules/python.mdx index db0f89b9b..e6e21507d 100644 --- a/pages/custom-query-modules/python.mdx +++ b/pages/custom-query-modules/python.mdx @@ -7,11 +7,79 @@ import { Callout } from 'nextra/components' # How to create a query module in Python -The [Python API](/custom-query-modules/python/python-api) provided by Memgraph -lets you develop query modules. It is accompanied by the [mock -API](/custom-query-modules/python/mock-python-api), which makes it possible to -develop and test query modules for Memgraph without having to run a Memgraph -instance. +The [Python API](/custom-query-modules/python/python-api) +provided by Memgraph lets you develop query modules. It is accompanied by the +[mock +API](/custom-query-modules/python/mock-python-api), +which makes it possible to develop and test query modules for Memgraph without +having to run a Memgraph instance. + +## Python API + +Python API is defined in the `mgp` module that can be found in the Memgraph +installation directory `/usr/lib/memgraph/python_support`. In essence, it is a +wrapper around the C API. If you wish to write your own query modules using the +Python API, you need to have Python version `3.5.0` or above installed. + +For more information, check the [Python API reference +guide](/custom-query-modules/python/python-api).
+We also made [an example module](/custom-query-modules/python/python-example) to +help you start developing your own modules. + +You can [develop query modules in Python from Memgraph +Lab](/custom-query-modules/python/implement-custom-query-module-in-python) (v2.0 +and newer). Just navigate to **Query Modules** and click on **New Module** to +start. + +![](/pages/custom-query-modules/memgraph_lab_query_modules.png) + +Custom modules developed via Memgraph Lab are located at +`/var/lib/memgraph/internal_modules`. + +### Mock Python API + +The [mock Python query module API](/custom-query-modules/python/mock-python-api) +enables you to develop and test query modules for Memgraph without having to run +a Memgraph instance by simulating its behavior. As the mock API is compatible +with the [Python API](/custom-query-modules/python/python-api), you can add +modules developed with it to Memgraph as-is, without modifying the code. + +For more information and examples, check the +[mock Python API reference guide](/custom-query-modules/python/mock-python-api). + +## Install external Python libraries + +It is possible to install Python libraries that are not already included with +Memgraph installation. For example, to install `pandas` with Memgraph running +inside a Docker container, run the following command in the terminal: + +```console +docker exec -i -u root bash -c "apt install -y python3-pip && +pip install pandas" +``` + +Don't forget to replace `` with the appropriate value, which you can +find by running `docker ps` command in the terminal. + +If you are starting Memgraph with [Docker +Compose](/getting-started/install-memgraph/docker-compose), write the following +commands in the Dockerfile: + +``` +FROM memgraph/memgraph:latest + +USER root + +RUN apt install -y python3-pip +RUN pip install pandas + +USER memgraph +``` + +It is important that you install Python library as a `root` user, rather than +the default `memgraph` user. + +## Example In this tutorial, we will learn how to develop a query module in Python on the example of the **random walk algorithm**. @@ -20,22 +88,12 @@ Python on the example of the **random walk algorithm**. There are three options for installing and working with Memgraph MAGE: -1. **Pulling the `memgraph/memgraph-mage` image**: check the `Docker Hub` - [installation guide](/advanced-algorithms/install-mage). -2. **Building a Docker image from the MAGE repository**: check the `Docker - build` [installation guide](/advanced-algorithms/install-mage). -3. **Building MAGE from source**: check the `Build from source on Linux` - [installation guide](/advanced-algorithms/install-mage). +- [install MAGE development environment from Docker Hub](/custom-query-modules#install-mage-with-docker-hub) +- [build MAGE using the `docker build` command](/custom-query-modules#install-mage-with-docker-build) +- [build MAGE from source](/custom-query-modules#install-mage-from-source) ## Developing a module - - -These steps are the same for all MAGE installation options (_Docker Hub_, -_Docker build_ and _Build from source on Linux_). - - - Position yourself in the **MAGE repository** you cloned earlier. Specifically, go in the `python` subdirectory and create a new file named `random_walk.py`. @@ -91,12 +149,6 @@ Memgraph it's a `read` procedure, meaning it won't make changes to the graph. The path is created from the `start` node, and edges are appended to it iteratively. - -If you need an additional Python library that is not already installed with -Memgraph, check out our [guide on how to install -it](/custom-query-modules/manage-query-modules). - - ### Terminate procedure execution Just as the execution of a Cypher query can be terminated with [`TERMINATE @@ -135,7 +187,7 @@ to the session where the procedure was originally run. ## Importing, querying and testing a module Now in order to import, query and test a module, check out the [following -page](/advanced-algorithms/run-algorithms). +page](/manage-query-modules). Feel free to create an issue or open a pull request on our [GitHub repo](https://github.com/memgraph/mage) to speed up the development.
@@ -156,11 +208,18 @@ accompanied by examples. ## Managing Memgraph's Python environment -After some time, any production system requires updates for the packages it uses. For example, when developing a new query module that requires the latest `networkx` version. +After some time, any production system requires updates for the packages it +uses. For example, when developing a new query module that requires the latest +`networkx` version. -If Memgraph is already deployed somewhere with an installed `networkx` package, you would probably like to use some package manager, e.g. pip or conda, to delete the old `networkx`, and install a new `networkx` package. You definitely wouldn't want to redeploy the whole Memgraph just because of one Python package. +If Memgraph is already deployed somewhere with an installed `networkx` package, +you would probably like to use some package manager, e.g. pip or conda, to +delete the old `networkx`, and install a new `networkx` package. You definitely +wouldn't want to redeploy the whole Memgraph just because of one Python package. -However, Python caches all modules, packages and the compiled bytecode, so this procedure cannot work out of the box. So after installing the new package, you need to use the utility procedure `mg.load_all()`. +However, Python caches all modules, packages and the compiled bytecode, so this +procedure cannot work out of the box. So after installing the new package, you +need to use the utility procedure `mg.load_all()`. So the whole process looks like this: