From 43fd0a2577f22b3bdaf339a23b920cfa5e9aa9a2 Mon Sep 17 00:00:00 2001 From: Vlasta Date: Mon, 14 Aug 2023 16:17:51 +0200 Subject: [PATCH 1/7] custom-procedures --- pages/custom-query-modules.mdx | 424 ++++++------------ pages/custom-query-modules/c.mdx | 19 + pages/custom-query-modules/cpp.mdx | 35 +- .../manage-query-modules.mdx | 96 ++++ pages/custom-query-modules/python.mdx | 117 +++-- 5 files changed, 366 insertions(+), 325 deletions(-) create mode 100644 pages/custom-query-modules/c.mdx diff --git a/pages/custom-query-modules.mdx b/pages/custom-query-modules.mdx index ffeb066e0..a102bbefb 100644 --- a/pages/custom-query-modules.mdx +++ b/pages/custom-query-modules.mdx @@ -1,20 +1,23 @@ import { Callout } from 'nextra/components' +import { Steps } from 'nextra/components' # Custom query modules -Memgraph supports extending the query language with user-written procedures in -**C**, **C++**, **Python**, and **Rust**. These procedures are grouped into -modules - **query modules** files (either `*.so` or `*.py` files). +Many [graph algorithms and uitility +procedures](/advanced-algorithms/available-algorithms) have already been +developed and are available as part of the MAGE library you can add to your +Memgraph installation. The library is already included if you are using Memgraph +Platform or Memgraph MAGE Docker images to run Memgraph. -Some query modules are built-in, and others, like those that can help you solve -complex graph issues, are available as part of the MAGE library you can add to -your Memgraph installation. The library is already included if you are using -Memgraph Platform or Memgraph MAGE Docker images to run Memgraph. +But, you can also implement custom query modules. Memgraph supports extending +the query language with user-written procedures in **C**, **C++**, **Python**, +and **Rust**. These procedures are grouped into modules - **query module** files +(either `*.so` or `*.py` files). -You can also implement custom query modules. Every single Memgraph installation -comes with the `example.so` and `py_example.py` query modules located in the -`/usr/lib/memgraph/query_modules` directory. They were provided as examples of -query modules for you to examine and learn from. +Every single Memgraph installation comes with the `example.so` and +`py_example.py` query modules located in the `/usr/lib/memgraph/query_modules` +directory. They were provided as examples of query modules for you to examine +and learn from. Each query module file corresponds to one query module, and file names are mapped as query module names. For example, `example.so` will be mapped as @@ -23,336 +26,183 @@ each module file has a procedure called `procedure` defined, those procedures would be mapped in the Cypher query language as `example.procedure()` and `py_example.procedure()` respectively. -Regardless of where they come from and who wrote them, all modules need to be -loaded into Memgraph so that they can be called while querying the database. -They are either loaded automatically when Memgraph starts or manually if they -were added while Memgraph was already running. +You can develop custom procedures from Memgraph itself. -You can also inspect and develop query modules in Memgraph Lab (v2.0 and newer). -Just navigate to **Query Modules**. +## Develop custom query modules -Screenshot of Query Modules from Memgraph Lab -![](/pages/custom-query-modules/memgraph_lab_query_modules.png) - - -Once you start Memgraph, it will attempt to load query modules from all *.so and -*.py files from the default directories. MAGE modules are located at -`/usr/lib/memgraph/query_modules` and custom modules developed via Memgraph Lab at -`/var/lib/memgraph/internal_modules`. - -Memgraph provides public APIs for writing custom query modules in Python, C and -C++. - -## Python API - -Python API is defined in the `mgp` module that can be found in the Memgraph -installation directory `/usr/lib/memgraph/python_support`. In essence, it is a -wrapper around the C API. If you wish to write your own query modules using the -Python API, you need to have Python version `3.5.0` or above installed. - -For more information, check the [Python API reference -guide](/custom-query-modules/python/python-api).
-We also made [an example module](/custom-query-modules/python/python-example) to -help you start developing your own modules. - -You can develop query modules in Python from Memgraph Lab (v2.0 and newer). Just -navigate to **Query Modules** and click on **New Module** to start. +If you want to develop Python procedures you can do so from [Memgraph +Lab](/data-visualization), the visual user interface. The files will be saved at +`/var/lib/memgraph/internal_modules`. Just navigate to **Query Modules** and +select *+ New Module*. ![](/pages/custom-query-modules/memgraph_lab_query_modules.png) -Custom modules developed via Memgraph Lab are located at -`/var/lib/memgraph/internal_modules`. - - -If you need an additional Python library not included with Memgraph, check out -[the guide on how to install -it](/memgraph/how-to-guides/query-modules#how-to-install-external-python-libraries). - - -### Mock Python API - -The [mock Python query module API](/custom-query-modules/python/mock-python-api) enables you to -develop and test query modules for Memgraph without having to run a Memgraph -instance by simulating its behavior. As the mock API is compatible with the -[Python API](/custom-query-modules/python/python-api), -you can add modules developed with it to Memgraph as-is, without modifying the -code. - -For more information and examples, check the -[mock Python API reference guide](/custom-query-modules/python/mock-python-api). +If you don't won't to develop from Memgraph Lab, you can: +- [install MAGE development environment from Docker Hub](#install-mage-with-docker-hub) +- [build MAGE using the `docker build` command](#install-mage-with-docker-build) +- [build MAGE from source](#install-mage-from-source) -## C API +then select a language you want to develop in. -C API modules need to be compiled to a shared library so that they can be loaded -when Memgraph starts. This means that you can write the procedures in any -programming language that can work with C and be compiled to the ELF shared -library format (`.so`). `mg_procedure.h` that can be found in Memgraph -installation directory `/usr/include/memgraph` contains declarations of all -functions that can be used to implement a query module procedure. To compile the -module, you will have to pass the appropriate flags to the compiler, for -example, `clang`: +## Install MAGE with Docker Hub -```plaintext -clang -Wall -shared -fPIC -I /usr/include/memgraph example.c -o example.so -``` - -For more information, check the [C API reference -guide](/custom-query-modules/c/c-api).
-We also made [an example module](/custom-query-modules/c/c-example) to help you -start developing your own modules. - -## C++ API - -C++ API modules, just like C API modules, need to be compiled to a shared -library so that they can be loaded when Memgraph starts. This is done much in -the same way as with C API modules. - -For more information, check the [C++ API reference -guide](/custom-query-modules/cpp/cpp-api).
-We also made [an example -module](/custom-query-modules/cpp/cpp-example) -to help you start developing your own modules. - - -If you need an additional Python library not included with Memgraph, check out -[the guide on how to install -it](/custom-query-modules/manage-query-modules). - - -## Install MAGE - -You can also download a MAGE image equipped for development inside of Docker -containers: +Download the MAGE image equipped for development inside of Docker containers: ```shell docker run -p 7687:7687 memgraph/memgraph-mage:1.1-dev ``` By running this command, you will get an image with the following tools -installed: Python3, Rust, Clang, Make, and CMake. This way, you can copy files -to the container, build them inside and import query modules in Memgraph. - -If you want to develop your own query modules, be sure to check the [Development -process for MAGE with -Docker](https://github.com/memgraph/mage#developing-mage-with-docker). +installed: `Python3`, `Rust`, `Clang`, `Make`, and `CMake`. This way, you can +copy files to the container, build them inside and load query modules into +Memgraph. +Develop query modules in [Python](/custom-query-modules/python), +[C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or +[Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). ----- +## Install MAGE with Docker build -## This is something taken from Install MAGE that seems it's used for building custom query_modules + -## This is when building MAGE from source with Linux - +Create a Docker image directly from the [MAGE GitHub +repository](https://github.com/memgraph/mage), instead of pulling it from the +Docker Hub. -Be sure you cloned the `mage` GitHub repository using the `--recurse-submodules` flag since it has incorporated Memgraph inside: - -```shell -git clone --recurse-submodules https://github.com/memgraph/mage.git -``` +### Download image or clone the repo -if you didn't, you can run following command to update submodules: - -```shell -git submodule update --init --recursive -``` - - - -**2.** Start Memgraph and enjoy **MAGE**! - - -Query modules are loaded into Memgraph on startup, so if your instance -was already running you will need to execute the following query inside one of -[querying platforms](https://memgraph.com/docs/memgraph/connect-to-memgraph) to -load them: - -``` -CALL mg.load_all(); -``` - -If your changes are not loaded, make sure to restart the instance by running -`systemctl stop memgraph` and `systemctl start memgraph`. - -If you want to find out more about loading query modules, visit [this -guide](/usage/loading-modules.md). - - - -### Advanced configuration - -#### 1. Set a different `query_modules` directory - -The `setup` script can set your local `mage/dist` directory or **any** other -directory as the **default** one in the Memgraph configuration file (flag -`--query-modules-directory` defined in `/etc/memgraph/memgraph.conf`). There are -a few options: - -**1.** Set `` as the **default** one: - -``` -python3 setup modules_storage -p -``` - -This way Memgraph will be looking for query modules inside ``. - - - -Don't forget to copy the aforementioned files from `mage/dist` to -``. - - - -**2.** Set `/mage/dist` as the **default** one: - -``` -python3 setup modules_storage -``` - -If the **default** directory is `mage/dist` then you don't need to copy `*.so` -and `*.py` files from the `mage/dist` directory -to`/usr/lib/memgraph/query_modules` every time you run `build`. - -## Docker build - -This way, you will create a Docker image directly from the [MAGE Github -repository](https://github.com/memgraph/mage) and won't have to pull it from -Docker Hub. You can: - -- download a [specific release](https://github.com/memgraph/mage/releases) from - the MAGE repository or -- clone the [repository](https://github.com/memgraph/mage) for the latest - version. - -If you downloaded a specific release, skip the first step. - -## Installing MAGE - -**1.** Download the MAGE source code from -**[GitHub](https://github.com/memgraph/mage)**: +Download a [specific release](https://github.com/memgraph/mage/releases) from +the MAGE repository or clone the [repository](https://github.com/memgraph/mage) for the latest +version. ```shell git clone --recurse-submodules https://github.com/memgraph/mage.git && cd mage ``` -**2.** Build the **MAGE** tagged Docker image with the following command: +### Build the MAGE tagged Docker image + +Run the the following command: ```shell docker build -t memgraph-mage . ``` -**3.** Start Memgraph-MAGE with the following command: +### Start Memgraph MAGE + +Run the following command: ```shell docker run --rm -p 7687:7687 --name mage memgraph-mage ``` - - -Now you can query Memgraph with any of the querying platforms like [Memgraph -Lab](https://memgraph.com/product/lab) or -[mgconsole](https://github.com/memgraph/mgconsole). - -If you made any changes while the **MAGE** Docker container was running, you -would need to stop the Docker container and rebuild the whole image. If you -don't want to repeat these steps each time, be sure to check the [Development -process for MAGE with -Docker](https://github.com/memgraph/mage#developing-mage-with-docker). - - - -## Developing MAGE with Docker - -When developing your query module, you need to load it inside Memgraph running -inside the Docker container. You can do that by [rebuilding the whole MAGE -image](#1-rebuild-the-whole-mage-image) or by [building it inside the Docker -container](#2-build-inside-the-docker-container). - -### 1. Rebuild the whole MAGE image - -This command will trigger the rebuild of the whole Docker image. Make sure that -you have added Python requirements inside `python/requirements.txt` file. - -**1.** Firstly, do the build of the **MAGE** image: - -``` -docker build -t memgraph-mage . -``` - -**2.** Now, start `memgraph-mage` image with the following command and enjoy -**your** own **MAGE**: - -``` -docker run --rm -p 7687:7687 --name mage memgraph-mage +### Develop modules + +Develop query modules in [Python](/custom-query-modules/python), +[C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or +[Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). + + + +## Install MAGE from source + +Follow the steps if you want to use the MAGE library with [installed Linux based +Memgraph package](/getting-started/install-memgraph) and create custom query modules. + + + +### Install dependencies + +To build from source, you will need: +- Python3 +- Make +- CMake +- Clang +- UUID +- [Rust and Cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html) + +### Set up the machine + +Run the following commands: + +```bash +sudo apt-get update && apt-get install -y \ + libcurl4 `memgraph` \ + libpython${PY_VERSION} `memgraph` \ + libssl-dev `memgraph` \ + openssl `memgraph` \ + build-essential `mage-memgraph` \ + cmake `mage-memgraph` \ + curl `mage-memgraph` \ + g++ `mage-memgraph` \ + python3 `mage-memgraph` \ + python3-pip `mage-memgraph` \ + python3-setuptools `mage-memgraph` \ + python3-dev `mage-memgraph` \ + clang `mage-memgraph` \ + git `mage-memgraph` \ + --no-install-recommends \ + && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* ``` -### 2. Build inside the Docker container +### Download the MAGE source code -You can build a **MAGE** Docker image equipped for development. `Rust`, `Clang`, -`Python3-pip`, and everything else necessary for development will still be -inside the running container. This means that you can copy the **MAGE** -repository to the container and do the build inside the `mage` container. There -is no need to do the whole Docker image build again. - -**1.** To create `dev` **MAGE** image, run the following command: +Clone the [MAGE source code](https://github.com/memgraph/mage) from GitHub: ``` -docker build --target dev -t memgraph-mage:dev . +git clone --recurse-submodules https://github.com/memgraph/mage.git ``` -**2.** Then run the image with the following command: - -``` -docker run --rm -p 7687:7687 --name mage memgraph-mage:dev -``` -**3.** Next, copy the files inside the container and do the build: +### Run the `setup` script -**a)** First, you need to copy the files to the container named `mage` +Run the following command: +```shell +python3 setup build -p /usr/lib/memgraph/query_modules ``` -docker cp . mage:/mage/ -``` - -**b)** Then, you need to position yourself inside the container as root: -``` -docker exec -u root -it mage /bin/bash -``` +The script will generate a `dist` directory with all the needed files. - +It will also copy the contents of the newly created `dist` directory to +`/usr/lib/memgraph/query_modules`. Memgraph loads algorithms and modules from +this directory. -Note: If you have done the build locally, make sure to delete the directory -`cpp/build` because you might be dealing with different `architectures` or -problems with `CMakeCache.txt`. To delete it, run: +If something isn't installed properly, the `setup` script will stop the +installation process. If you have any questions, contact us on +**[Discord](https://discord.gg/memgraph).** -`rm -rf cpp/build` +#### Set a different `query_modules` directory - +The `setup` script can set your local `mage/dist` directory or any other +directory as the default one by changing the value of the +`--query-modules-directory` flag in the `/etc/memgraph/memgraph.conf`, +Memgraph's configuration file. -**c)** After that, run build and copy `mage/dist` to -`/usr/lib/memgraph/query_modules`: +By setting the `/mage/dist` as the default directory you don't need to copy +`*.so` and `*.py` files from the `mage/dist` directory +to`/usr/lib/memgraph/query_modules` every time you run `build`: ``` -python3 setup build -p /usr/lib/memgraph/query_modules/ +python3 setup modules_storage ``` -**d)** Everything should be ready, and you can run the following command to exit -the container: +By setting `` as the default one, Memgraph will be looking for +query modules inside ``, instead of `/usr/lib/memgraph/query_modules`: ``` -exit +python3 setup modules_storage -p ``` - +Don't forget to copy the aforementioned files from `mage/dist` to +``. -Note that query modules are loaded into Memgraph on startup, so if your instance -was already running, you would need to execute the following query inside one of -the [querying platforms](https://memgraph.com/docs/memgraph/connect-to-memgraph) -to load them: +### Develop modules -`CALL mg.load_all();` +Develop query modules in [Python](/custom-query-modules/python), +[C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or +[Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). - + ## Memgraph X cuGraph @@ -364,7 +214,7 @@ us](https://memgraph.com/community) about it. -### Building MAGE with NVIDIA cuGraph locally with Docker +### Building MAGE with NVIDIA cuGraph locally with Docker - what is this 1. Download the MAGE source code from [GitHub](https://github.com/memgraph/mage): @@ -395,4 +245,18 @@ the container and rebuild the image. For a workaround, check [Development process for MAGE with Docker](/installation/docker-build.md#developing-mage-with-docker). - \ No newline at end of file + + +## Load custom query modules + +Once you start Memgraph, it will attempt to load query modules from all *.so and +*.py files from the default directories. MAGE modules are located at +`/usr/lib/memgraph/query_modules` by default, but Memgraph will also load any +modules located at `/var/lib/memgraph/internal_modules`. + +If the files were added while Memgraph was already running, they can be manually +loaded into Memgraph using the following query: + +``` +CALL mg.load_all(); +``` \ No newline at end of file diff --git a/pages/custom-query-modules/c.mdx b/pages/custom-query-modules/c.mdx new file mode 100644 index 000000000..edc4ace4b --- /dev/null +++ b/pages/custom-query-modules/c.mdx @@ -0,0 +1,19 @@ +# How to create a query module in C++ + +C API modules need to be compiled to a shared library so that they can be loaded +when Memgraph starts. This means that you can write the procedures in any +programming language that can work with C and be compiled to the ELF shared +library format (`.so`). `mg_procedure.h` that can be found in Memgraph +installation directory `/usr/include/memgraph` contains declarations of all +functions that can be used to implement a query module procedure. To compile the +module, you will have to pass the appropriate flags to the compiler, for +example, `clang`: + +```plaintext +clang -Wall -shared -fPIC -I /usr/include/memgraph example.c -o example.so +``` + +For more information, check the [C API reference +guide](/custom-query-modules/c/c-api).
+We also made [an example module](/custom-query-modules/c/c-example) to help you +start developing your own modules. \ No newline at end of file diff --git a/pages/custom-query-modules/cpp.mdx b/pages/custom-query-modules/cpp.mdx index e7dda836d..0a8663a04 100644 --- a/pages/custom-query-modules/cpp.mdx +++ b/pages/custom-query-modules/cpp.mdx @@ -2,31 +2,34 @@ import { Callout } from 'nextra/components' # How to create a query module in C++ +## C++ API + Query modules can be implemented using the [C++ -API](/memgraph/reference-guide/query-modules/api/cpp-api) -provided by Memgraph with automatic memory management. -In this tutorial, we will learn how to develop a query -module in C++ on the example of the **random walk algorithm**. +API](/memgraph/reference-guide/query-modules/api/cpp-api) provided by Memgraph +with automatic memory management. C++ API modules, just like C API modules, need +to be compiled to a shared library so that they can be loaded when Memgraph +starts. This is done much in the same way as with C API modules. + +For more information, check the [C++ API reference +guide](/custom-query-modules/cpp/cpp-api).
+We also made [an example module](/custom-query-modules/cpp/cpp-example) to help +you start developing your own modules. + +## Example + +In this tutorial, we will learn how to develop a query module in C++ on the +example of the **random walk algorithm**. ## Prerequisites There are three options for installing and working with Memgraph MAGE: -1. **Pulling the `memgraph/memgraph-mage` image**: check the `Docker Hub` - [installation guide](/installation/docker-hub.md). -2. **Building a Docker image from the MAGE repository**: check the `Docker - build` [installation guide](/installation/docker-build.md). -3. **Building MAGE from source**: check the `Build from source on Linux` - [installation guide](/installation/source.md). +- [install MAGE development environment from Docker Hub](/custom-query-modules#install-mage-with-docker-hub) +- [build MAGE using the `docker build` command](/custom-query-modules#install-mage-with-docker-build) +- [build MAGE from source](/custom-query-modules#install-mage-from-source) ## Developing a module - - -These instructions are the same for every MAGE installation option: _Docker -Hub_, _Docker build_ and _Build from source on Linux_. - - Position yourself in the **MAGE repository** you cloned earlier. Once you are there, enter the `cpp` subdirectory and create a new directory called diff --git a/pages/custom-query-modules/manage-query-modules.mdx b/pages/custom-query-modules/manage-query-modules.mdx index b748b65d9..9c2c91d30 100644 --- a/pages/custom-query-modules/manage-query-modules.mdx +++ b/pages/custom-query-modules/manage-query-modules.mdx @@ -1,5 +1,101 @@ import { Callout } from 'nextra/components' +## This is something to load the modules + +## Developing MAGE with Docker + +When developing your query module, you need to load it inside Memgraph running +inside the Docker container. You can do that by [rebuilding the whole MAGE +image](#1-rebuild-the-whole-mage-image) or by [building it inside the Docker +container](#2-build-inside-the-docker-container). + +### 1. Rebuild the whole MAGE image + +This command will trigger the rebuild of the whole Docker image. Make sure that +you have added Python requirements inside `python/requirements.txt` file. + +**1.** Firstly, do the build of the **MAGE** image: + +``` +docker build -t memgraph-mage . +``` + +**2.** Now, start `memgraph-mage` image with the following command and enjoy +**your** own **MAGE**: + +``` +docker run --rm -p 7687:7687 --name mage memgraph-mage +``` + +### 2. Build inside the Docker container + +You can build a **MAGE** Docker image equipped for development. `Rust`, `Clang`, +`Python3-pip`, and everything else necessary for development will still be +inside the running container. This means that you can copy the **MAGE** +repository to the container and do the build inside the `mage` container. There +is no need to do the whole Docker image build again. + +**1.** To create `dev` **MAGE** image, run the following command: + +``` +docker build --target dev -t memgraph-mage:dev . +``` + +**2.** Then run the image with the following command: + +``` +docker run --rm -p 7687:7687 --name mage memgraph-mage:dev +``` + +**3.** Next, copy the files inside the container and do the build: + +**a)** First, you need to copy the files to the container named `mage` + +``` +docker cp . mage:/mage/ +``` + +**b)** Then, you need to position yourself inside the container as root: + +``` +docker exec -u root -it mage /bin/bash +``` + + + +Note: If you have done the build locally, make sure to delete the directory +`cpp/build` because you might be dealing with different `architectures` or +problems with `CMakeCache.txt`. To delete it, run: + +`rm -rf cpp/build` + + + +**c)** After that, run build and copy `mage/dist` to +`/usr/lib/memgraph/query_modules`: + +``` +python3 setup build -p /usr/lib/memgraph/query_modules/ +``` + +**d)** Everything should be ready, and you can run the following command to exit +the container: + +``` +exit +``` + + + +Note that query modules are loaded into Memgraph on startup, so if your instance +was already running, you would need to execute the following query inside one of +the [querying platforms](https://memgraph.com/docs/memgraph/connect-to-memgraph) +to load them: + +`CALL mg.load_all();` + + + # Manage query modules The following page describes how query modules are loaded into Memgraph and diff --git a/pages/custom-query-modules/python.mdx b/pages/custom-query-modules/python.mdx index 93ff47ec1..2467c7633 100644 --- a/pages/custom-query-modules/python.mdx +++ b/pages/custom-query-modules/python.mdx @@ -2,11 +2,79 @@ import { Callout } from 'nextra/components' # How to create a query module in Python -The [Python API](/memgraph/reference-guide/query-modules/api/python-api) +The [Python API](/custom-query-modules/šython/python-api) provided by Memgraph lets you develop query modules. It is accompanied by the -[mock API](https://memgraph.com/docs/memgraph/reference-guide/query-modules/api/mock-python-api), which -makes it possible to develop and test query modules for Memgraph without having -to run a Memgraph instance. +[mock +API](/custom-query-modules/python/mock-python-api), +which makes it possible to develop and test query modules for Memgraph without +having to run a Memgraph instance. + +## Python API + +Python API is defined in the `mgp` module that can be found in the Memgraph +installation directory `/usr/lib/memgraph/python_support`. In essence, it is a +wrapper around the C API. If you wish to write your own query modules using the +Python API, you need to have Python version `3.5.0` or above installed. + +For more information, check the [Python API reference +guide](/custom-query-modules/python/python-api).
+We also made [an example module](/custom-query-modules/python/python-example) to +help you start developing your own modules. + +You can [develop query modules in Python from Memgraph +Lab](/custom-query-modules/python/implement-custom-query-module-in-python) (v2.0 +and newer). Just navigate to **Query Modules** and click on **New Module** to +start. + +![](/pages/custom-query-modules/memgraph_lab_query_modules.png) + +Custom modules developed via Memgraph Lab are located at +`/var/lib/memgraph/internal_modules`. + +### Mock Python API + +The [mock Python query module API](/custom-query-modules/python/mock-python-api) +enables you to develop and test query modules for Memgraph without having to run +a Memgraph instance by simulating its behavior. As the mock API is compatible +with the [Python API](/custom-query-modules/python/python-api), you can add +modules developed with it to Memgraph as-is, without modifying the code. + +For more information and examples, check the +[mock Python API reference guide](/custom-query-modules/python/mock-python-api). + +## Install external Python libraries + +It is possible to install Python libraries that are not already included with +Memgraph installation. For example, to install `pandas` with Memgraph running +inside a Docker container, run the following command in the terminal: + +```console +docker exec -i -u root bash -c "apt install -y python3-pip && +pip install pandas" +``` + +Don't forget to replace `` with the appropriate value, which you can +find by running `docker ps` command in the terminal. + +If you are starting Memgraph with [Docker +Compose](/getting-started/install-memgraph/docker-compose), write the following +commands in the Dockerfile: + +``` +FROM memgraph/memgraph:latest + +USER root + +RUN apt install -y python3-pip +RUN pip install pandas + +USER memgraph +``` + +It is important that you install Python library as a `root` user, rather than +the default `memgraph` user. + +## Example In this tutorial, we will learn how to develop a query module in Python on the example of the **random walk algorithm**. @@ -15,22 +83,12 @@ Python on the example of the **random walk algorithm**. There are three options for installing and working with Memgraph MAGE: -1. **Pulling the `memgraph/memgraph-mage` image**: check the `Docker Hub` - [installation guide](/installation/docker-hub.md). -2. **Building a Docker image from the MAGE repository**: check the `Docker - build` [installation guide](/installation/docker-build.md). -3. **Building MAGE from source**: check the `Build from source on Linux` - [installation guide](/installation/source.md). +- [install MAGE development environment from Docker Hub](/custom-query-modules#install-mage-with-docker-hub) +- [build MAGE using the `docker build` command](/custom-query-modules#install-mage-with-docker-build) +- [build MAGE from source](/custom-query-modules#install-mage-from-source) ## Developing a module - - -These steps are the same for all MAGE installation options (_Docker Hub_, -_Docker build_ and _Build from source on Linux_). - - - Position yourself in the **MAGE repository** you cloned earlier. Specifically, go in the `python` subdirectory and create a new file named `random_walk.py`. @@ -86,12 +144,6 @@ Memgraph it's a `read` procedure, meaning it won't make changes to the graph. The path is created from the `start` node, and edges are appended to it iteratively. - -If you need an additional Python library that is not already installed with -Memgraph, check out our [guide on how to install -it](/memgraph/how-to-guides/query-modules#how-to-install-external-python-libraries). - - ### Terminate procedure execution Just as the execution of a Cypher query can be terminated with [`TERMINATE @@ -130,7 +182,7 @@ to the session where the procedure was originally run. ## Importing, querying and testing a module Now in order to import, query and test a module, check out the [following -page](/mage/how-to-guides/run-a-query-module). +page](/manage-query-modules). Feel free to create an issue or open a pull request on our [GitHub repo](https://github.com/memgraph/mage) to speed up the development.
@@ -139,23 +191,30 @@ Also, don't forget to throw us a star on GitHub. :star: ## Working with the mock API The -[mock Python API](https://memgraph.com/docs/memgraph/reference-guide/query-modules/api/mock-python-api) +[mock Python API](/custom-query-modules/python/mock-python-api) lets you develop and test query modules for Memgraph without having to run a Memgraph instance. As it’s compatible with the Python API you can add modules developed with it to Memgraph as-is, without having to refactor your code. The documentation on importing the mock API and running query modules with it is available -[here](https://memgraph.com/docs/memgraph/reference-guide/query-modules/api/mock-python-api#using-the-mock-api), +[here](/custom-query-modules/python/mock-python-api#using-the-mock-api), accompanied by examples. ## Managing Memgraph's Python environment -After some time, any production system requires updates for the packages it uses. For example, when developing a new query module that requires the latest `networkx` version. +After some time, any production system requires updates for the packages it +uses. For example, when developing a new query module that requires the latest +`networkx` version. -If Memgraph is already deployed somewhere with an installed `networkx` package, you would probably like to use some package manager, e.g. pip or conda, to delete the old `networkx`, and install a new `networkx` package. You definitely wouldn't want to redeploy the whole Memgraph just because of one Python package. +If Memgraph is already deployed somewhere with an installed `networkx` package, +you would probably like to use some package manager, e.g. pip or conda, to +delete the old `networkx`, and install a new `networkx` package. You definitely +wouldn't want to redeploy the whole Memgraph just because of one Python package. -However, Python caches all modules, packages and the compiled bytecode, so this procedure cannot work out of the box. So after installing the new package, you need to use the utility procedure `mg.load_all()`. +However, Python caches all modules, packages and the compiled bytecode, so this +procedure cannot work out of the box. So after installing the new package, you +need to use the utility procedure `mg.load_all()`. So the whole process looks like this: From 0d91a0e80563dec0e36fea8ea71a57b0f65ac392 Mon Sep 17 00:00:00 2001 From: Vlasta Date: Mon, 14 Aug 2023 18:48:46 +0200 Subject: [PATCH 2/7] something --- pages/custom-query-modules.mdx | 481 +++++++++++++----- .../manage-query-modules.mdx | 96 ---- 2 files changed, 356 insertions(+), 221 deletions(-) diff --git a/pages/custom-query-modules.mdx b/pages/custom-query-modules.mdx index a102bbefb..4770032f6 100644 --- a/pages/custom-query-modules.mdx +++ b/pages/custom-query-modules.mdx @@ -1,5 +1,6 @@ import { Callout } from 'nextra/components' import { Steps } from 'nextra/components' +import { Tabs, Tab } from 'nextra/components' # Custom query modules @@ -38,173 +39,403 @@ select *+ New Module*. ![](/pages/custom-query-modules/memgraph_lab_query_modules.png) If you don't won't to develop from Memgraph Lab, you can: -- [install MAGE development environment from Docker Hub](#install-mage-with-docker-hub) -- [build MAGE using the `docker build` command](#install-mage-with-docker-build) -- [build MAGE from source](#install-mage-from-source) +- [install MAGE development environment from Docker Hub] +- [build MAGE using the `docker build` command] +- [build MAGE from source] then select a language you want to develop in. -## Install MAGE with Docker Hub +## Install MAGE and import query modules -Download the MAGE image equipped for development inside of Docker containers: + -```shell -docker run -p 7687:7687 memgraph/memgraph-mage:1.1-dev -``` + -By running this command, you will get an image with the following tools -installed: `Python3`, `Rust`, `Clang`, `Make`, and `CMake`. This way, you can -copy files to the container, build them inside and load query modules into -Memgraph. + -Develop query modules in [Python](/custom-query-modules/python), -[C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or -[Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). + ### Download the MAGE image + + Run the following command to download the MAGE image equipped for development inside of Docker containers: -## Install MAGE with Docker build + ```shell + docker run -p 7687:7687 memgraph/memgraph-mage:version-dev + ``` - + By running this command, you will get an image with the following tools + installed: `Python3`, `Rust`, `Clang`, `Make`, and `CMake`. This way, you can + copy files to the container, build them inside and load query modules into + Memgraph. -Create a Docker image directly from the [MAGE GitHub -repository](https://github.com/memgraph/mage), instead of pulling it from the -Docker Hub. + ### Develop a query -### Download image or clone the repo + Develop query modules in [Python](/custom-query-modules/python), + [C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or + [Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). -Download a [specific release](https://github.com/memgraph/mage/releases) from -the MAGE repository or clone the [repository](https://github.com/memgraph/mage) for the latest -version. + ### Start the MAGE container -```shell -git clone --recurse-submodules https://github.com/memgraph/mage.git && cd mage -``` + Use the following command to start the MAGE container: -### Build the MAGE tagged Docker image + ```shell + docker run --rm -p 7687:7687 --name mage memgraph-mage:version-dev + ``` -Run the the following command: + Be sure to replace the version with the specific version, for example: -```shell -docker build -t memgraph-mage . -``` + ```shell + docker run --rm -p 7687:7687 --name mage memgraph-mage:1.4-dev + ``` -### Start Memgraph MAGE + ### Copy the files to the container -Run the following command: + Copy the files to the container named mage: -```shell -docker run --rm -p 7687:7687 --name mage memgraph-mage -``` + ```shell + docker cp . mage:/mage/ + ``` -### Develop modules - -Develop query modules in [Python](/custom-query-modules/python), -[C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or -[Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). - - - -## Install MAGE from source - -Follow the steps if you want to use the MAGE library with [installed Linux based -Memgraph package](/getting-started/install-memgraph) and create custom query modules. - - - -### Install dependencies - -To build from source, you will need: -- Python3 -- Make -- CMake -- Clang -- UUID -- [Rust and Cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html) - -### Set up the machine - -Run the following commands: - -```bash -sudo apt-get update && apt-get install -y \ - libcurl4 `memgraph` \ - libpython${PY_VERSION} `memgraph` \ - libssl-dev `memgraph` \ - openssl `memgraph` \ - build-essential `mage-memgraph` \ - cmake `mage-memgraph` \ - curl `mage-memgraph` \ - g++ `mage-memgraph` \ - python3 `mage-memgraph` \ - python3-pip `mage-memgraph` \ - python3-setuptools `mage-memgraph` \ - python3-dev `mage-memgraph` \ - clang `mage-memgraph` \ - git `mage-memgraph` \ - --no-install-recommends \ - && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* -``` + ### Enter the container -### Download the MAGE source code + Position yourself inside the container as `root`: -Clone the [MAGE source code](https://github.com/memgraph/mage) from GitHub: + ```shell + docker exec -u root -it mage /bin/bash + ``` + -``` -git clone --recurse-submodules https://github.com/memgraph/mage.git -``` + If you performed the build locally, make sure to delete the cpp/build directory because you might be dealing with different architectures or problems with `CMakeCache.txt`. + To delete it, run: -### Run the `setup` script + ```shell + rm -rf cpp/build + ``` -Run the following command: + -```shell -python3 setup build -p /usr/lib/memgraph/query_modules -``` + ### Build MAGE -The script will generate a `dist` directory with all the needed files. + Build MAGE with the option to copy executables from `mage/dist` to `/usr/lib/memgraph/query_modules`: -It will also copy the contents of the newly created `dist` directory to -`/usr/lib/memgraph/query_modules`. Memgraph loads algorithms and modules from -this directory. + ```shell + python3 setup build -p /usr/lib/memgraph/query_modules/ + ``` -If something isn't installed properly, the `setup` script will stop the -installation process. If you have any questions, contact us on -**[Discord](https://discord.gg/memgraph).** + ### Exit the container -#### Set a different `query_modules` directory + Everything should be ready to exit the container and load the query modules: -The `setup` script can set your local `mage/dist` directory or any other -directory as the default one by changing the value of the -`--query-modules-directory` flag in the `/etc/memgraph/memgraph.conf`, -Memgraph's configuration file. + ```shell + exit + ``` -By setting the `/mage/dist` as the default directory you don't need to copy -`*.so` and `*.py` files from the `mage/dist` directory -to`/usr/lib/memgraph/query_modules` every time you run `build`: + -``` -python3 setup modules_storage + + + + Install MAGE with Docker build + + Create a Docker image directly from the [MAGE GitHub + repository](https://github.com/memgraph/mage), instead of pulling it from the + Docker Hub. + + + + ### Download image or clone the repo + + Download a [specific release](https://github.com/memgraph/mage/releases) from + the MAGE repository or clone the [repository](https://github.com/memgraph/mage) for the latest + version. + + ```shell + git clone --recurse-submodules https://github.com/memgraph/mage.git && cd mage + ``` + + ### Build the MAGE tagged Docker image + + Run the the following command: + + ```shell + docker build -t memgraph-mage . + ``` + + ### Start Memgraph MAGE + + Run the following command: + + ```shell + docker run --rm -p 7687:7687 --name mage memgraph-mage + ``` + + ### Develop modules + + Develop query modules in [Python](/custom-query-modules/python), + [C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or + [Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). + + ### Create the `dev` MAGE image + + To create the `dev` MAGE image, run the following command: + + ```shell + docker build --target dev -t memgraph-mage:dev . + ``` + + ### Start the container + + Start the container with the following command: + + ```shell + docker run --rm -p 7687:7687 --name mage memgraph-mage:dev + ``` + + ### Rebuild everything + + If you make any changes to the module, you can stop the container and do a + rebuild. Additionally, if you don't want to rebuild everything, continue with the next step. + + ### Copy the files to the container + + Copy the files to the container named `mage`: + + ```shell + docker cp . mage:/mage/ + ``` + + ### Enter the container + + Position yourself inside the container as `root`: + + ```shell + docker exec -u root -it mage /bin/bash + ``` + + + If you performed the build locally, make sure to delete the cpp/build directory because you might be dealing with different architectures or problems with `CMakeCache.txt`. + + To delete it, run: + + ```shell + rm -rf cpp/build + ``` + + + + ### Build MAGE + + Build MAGE with the option to copy executables from `mage/dist` to `/usr/lib/memgraph/query_modules`: + + ```shell + python3 setup build -p /usr/lib/memgraph/query_modules/ + ``` + + ### Exit the container + + Everything should be ready to exit the container and load the query modules: + + ```shell + exit + ``` + + + + + + Follow the steps if you want to use the MAGE library with [installed Linux based + Memgraph package](/getting-started/install-memgraph) and create custom query modules. + + + + ### Install dependencies + + To build from source, you will need: + - Python3 + - Make + - CMake + - Clang + - UUID + - [Rust and Cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html) + + ### Set up the machine + + Run the following commands: + + ```bash + sudo apt-get update && apt-get install -y \ + libcurl4 `memgraph` \ + libpython${PY_VERSION} `memgraph` \ + libssl-dev `memgraph` \ + openssl `memgraph` \ + build-essential `mage-memgraph` \ + cmake `mage-memgraph` \ + curl `mage-memgraph` \ + g++ `mage-memgraph` \ + python3 `mage-memgraph` \ + python3-pip `mage-memgraph` \ + python3-setuptools `mage-memgraph` \ + python3-dev `mage-memgraph` \ + clang `mage-memgraph` \ + git `mage-memgraph` \ + --no-install-recommends \ + && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* + ``` + + ### Download the MAGE source code + + Clone the [MAGE source code](https://github.com/memgraph/mage) from GitHub: + + ``` + git clone --recurse-submodules https://github.com/memgraph/mage.git + ``` + + + ### Run the `setup` script + + Run the following command: + + ```shell + python3 setup build -p /usr/lib/memgraph/query_modules + ``` + + The script will generate a `dist` directory with all the needed files. + + It will also copy the contents of the newly created `dist` directory to + `/usr/lib/memgraph/query_modules`. Memgraph loads algorithms and modules from + this directory. + + If something isn't installed properly, the `setup` script will stop the + installation process. If you have any questions, contact us on + **[Discord](https://discord.gg/memgraph).** + + #### Set a different `query_modules` directory + + The `setup` script can set your local `mage/dist` directory or any other + directory as the default one by changing the value of the + `--query-modules-directory` flag in the `/etc/memgraph/memgraph.conf`, + Memgraph's configuration file. + + By setting the `/mage/dist` as the default directory you don't need to copy + `*.so` and `*.py` files from the `mage/dist` directory + to`/usr/lib/memgraph/query_modules` every time you run `build`: + + ``` + python3 setup modules_storage + ``` + + By setting `` as the default one, Memgraph will be looking for + query modules inside ``, instead of `/usr/lib/memgraph/query_modules`: + + ``` + python3 setup modules_storage -p + ``` + + Don't forget to copy the aforementioned files from `mage/dist` to + ``. + + ### Develop modules + + Develop query modules in [Python](/custom-query-modules/python), + [C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or + [Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). + + ### Start Memgraph + + Make sure your Memgraph instance is running: + + ``` + sudo systemctl status memgraph.service + ``` + + ### Copy the query module + + Copy your developed query module to `/usr/lib/memgraph/query_modules` or your directory if you changed the default location of query modules: + + ```shell + python3 setup build -p /usr/lib/memgraph/query_modules + ``` + + + + + + + + +## Querying + +> Note that query modules are loaded into Memgraph on startup, so if your +> instance was already running, you would need to execute the following query +> inside one of the [querying +> platforms](https://docs.memgraph.com/memgraph/connect-to-memgraph) to load +> them: + +```cypher +CALL mg.load_all(); ``` -By setting `` as the default one, Memgraph will be looking for -query modules inside ``, instead of `/usr/lib/memgraph/query_modules`: +Lastly, run a query and test your module: +```cypher +MERGE (start:Node {id: 0})-[:RELATION]->(:Node {id: 1})-[:RELATION]->(:Node {id: 2}) +CALL random_walk.get(start, 2) YIELD path +RETURN path ``` -python3 setup modules_storage -p + +Since every query module is run as one transaction in Memgraph, the user can stop +the query module by [terminating the corresponding transaction](/memgraph/reference-guide/transactions). The user first needs +to find out the transaction ID using `SHOW TRANSACTIONS` command and then terminate it +using the `TERMINATE TRANSACTIONS ` command. + +## Testing + +Test decoupled parts of your code that don't depend on Memgraph like you would +in any other setting. E2e (end to end) tests, on the other hand, depend on +internal Memgraph data structures, like nodes and edges. After running Memgraph, +we need to prepare the testing environment on the host machine. Position +yourself in the mage directory you cloned from GitHub. The expected folder +structure for each module is the following: + +```plaintext +mage +└── e2e + └── random_walk_test + └── test_base + ├── input.cyp + └── test.yml ``` -Don't forget to copy the aforementioned files from `mage/dist` to -``. +`input.cyp` represents a Cypher script for entering the data into the database. +To simplify this tutorial, we'll leave the database empty. `test.yml` specifies +which test query should be run by the database and what should be the result or +exception. Create the files following the aforementioned directory structure. + +### input.cyp + +```cypher +MATCH (n) DETACH DELETE n; +``` -### Develop modules +### test.yml -Develop query modules in [Python](/custom-query-modules/python), -[C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or -[Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). +```shell +query: > + MATCH (start:Node {id: 0}) + CALL random_walk.get(start, 2) YIELD path + RETURN path + +output: [] +``` + +Lastly, run the e2e tests with python: + +```shell +python test_e2e +``` - +## I don't know what to do with this knowledge -## Memgraph X cuGraph +### Memgraph X cuGraph @@ -214,7 +445,7 @@ us](https://memgraph.com/community) about it. -### Building MAGE with NVIDIA cuGraph locally with Docker - what is this +#### Building MAGE with NVIDIA cuGraph locally with Docker - what is this 1. Download the MAGE source code from [GitHub](https://github.com/memgraph/mage): @@ -247,7 +478,7 @@ Docker](/installation/docker-build.md#developing-mage-with-docker). -## Load custom query modules +### Load custom query modules Once you start Memgraph, it will attempt to load query modules from all *.so and *.py files from the default directories. MAGE modules are located at diff --git a/pages/custom-query-modules/manage-query-modules.mdx b/pages/custom-query-modules/manage-query-modules.mdx index 9c2c91d30..b748b65d9 100644 --- a/pages/custom-query-modules/manage-query-modules.mdx +++ b/pages/custom-query-modules/manage-query-modules.mdx @@ -1,101 +1,5 @@ import { Callout } from 'nextra/components' -## This is something to load the modules - -## Developing MAGE with Docker - -When developing your query module, you need to load it inside Memgraph running -inside the Docker container. You can do that by [rebuilding the whole MAGE -image](#1-rebuild-the-whole-mage-image) or by [building it inside the Docker -container](#2-build-inside-the-docker-container). - -### 1. Rebuild the whole MAGE image - -This command will trigger the rebuild of the whole Docker image. Make sure that -you have added Python requirements inside `python/requirements.txt` file. - -**1.** Firstly, do the build of the **MAGE** image: - -``` -docker build -t memgraph-mage . -``` - -**2.** Now, start `memgraph-mage` image with the following command and enjoy -**your** own **MAGE**: - -``` -docker run --rm -p 7687:7687 --name mage memgraph-mage -``` - -### 2. Build inside the Docker container - -You can build a **MAGE** Docker image equipped for development. `Rust`, `Clang`, -`Python3-pip`, and everything else necessary for development will still be -inside the running container. This means that you can copy the **MAGE** -repository to the container and do the build inside the `mage` container. There -is no need to do the whole Docker image build again. - -**1.** To create `dev` **MAGE** image, run the following command: - -``` -docker build --target dev -t memgraph-mage:dev . -``` - -**2.** Then run the image with the following command: - -``` -docker run --rm -p 7687:7687 --name mage memgraph-mage:dev -``` - -**3.** Next, copy the files inside the container and do the build: - -**a)** First, you need to copy the files to the container named `mage` - -``` -docker cp . mage:/mage/ -``` - -**b)** Then, you need to position yourself inside the container as root: - -``` -docker exec -u root -it mage /bin/bash -``` - - - -Note: If you have done the build locally, make sure to delete the directory -`cpp/build` because you might be dealing with different `architectures` or -problems with `CMakeCache.txt`. To delete it, run: - -`rm -rf cpp/build` - - - -**c)** After that, run build and copy `mage/dist` to -`/usr/lib/memgraph/query_modules`: - -``` -python3 setup build -p /usr/lib/memgraph/query_modules/ -``` - -**d)** Everything should be ready, and you can run the following command to exit -the container: - -``` -exit -``` - - - -Note that query modules are loaded into Memgraph on startup, so if your instance -was already running, you would need to execute the following query inside one of -the [querying platforms](https://memgraph.com/docs/memgraph/connect-to-memgraph) -to load them: - -`CALL mg.load_all();` - - - # Manage query modules The following page describes how query modules are loaded into Memgraph and From 7c41bc45cfadb2c97b227eaf91ce365f4cfa6914 Mon Sep 17 00:00:00 2001 From: Vlasta Date: Wed, 16 Aug 2023 08:05:30 +0200 Subject: [PATCH 3/7] manage-qm --- .../manage-query-modules.mdx | 84 +++---------------- 1 file changed, 11 insertions(+), 73 deletions(-) diff --git a/pages/custom-query-modules/manage-query-modules.mdx b/pages/custom-query-modules/manage-query-modules.mdx index b748b65d9..44cf5bf07 100644 --- a/pages/custom-query-modules/manage-query-modules.mdx +++ b/pages/custom-query-modules/manage-query-modules.mdx @@ -2,76 +2,6 @@ import { Callout } from 'nextra/components' # Manage query modules -The following page describes how query modules are loaded into Memgraph and -called within a Cypher query. - -## Loading query modules - -Once you start Memgraph, it will attempt to load query modules from all `*.so` -and `*.py` files from the default (`/usr/lib/memgraph/query_modules` and -`/var/lib/memgraph/internal_modules`) directories. - -MAGE modules are located at -`/usr/lib/memgraph/query_modules` and custom modules developed via Memgraph Lab at -`/var/lib/memgraph/internal_modules`. - -Memgraph can load query modules from additional directories, if their path is -added to the `--query-modules-directory` flag in the main configuration file -(`/etc/memgraph/memgraph.conf`) or supplied as a command-line parameter (e.g. -when using Docker). - -If you are supplying the additional directory as a parameter, do not forget to -include the path to `/usr/lib/memgraph/query_modules`, otherwise queries from -that directory will not be loaded when Memgraph starts. - - - -When working with Docker and `memgraph-platform` image, you should pass -configuration flags inside of environment variables, for example: - -```terminal -docker run -p 7687:7687 -p 7444:7444 -p 3000:3000 -e MEMGRAPH="--query-modules-directory=/usr/lib/memgraph/query_modules,/usr/lib/memgraph/my_modules" memgraph/memgraph-platform` -``` - -If you are working with `memgraph` or `memgraph-mage` images you should pass -configuration options like this: - -```terminal -docker run -p 7687:7687 -p 7444:7444 memgraph/memgraph --query-modules-directory=/usr/lib/memgraph/query_modules,/usr/lib/memgraph/my_modules -``` - - - -If a certain query module was added while Memgraph was already running, you need -to load it manually using the `mg.load("module_name")` procedure within a query: - -```cypher -CALL mg.load("py_example"); -``` - -If there is no response (no error message), the load was successful. - -If you want to reload all existing modules and load any newly added ones, use -`mg.load_all()`: - -```cypher -CALL mg.load_all(); -``` - -If there is no response (no error message), the load was successful. - -You can check if the query module has been loaded by using the `mg.procedures()` -procedure within a query: - -```cypher -CALL mg.procedures() YIELD *; -``` - -Built-in utility query module (`mg`) contains procedures that enable you to -manage query modules files. - - -### This is from advanced algorithms, check if there is anything new Here ## Load procedures Once you start Memgraph, it will attempt to load query modules from all `*.so` @@ -142,7 +72,10 @@ syntax: CALL module.procedure([optional parameter], arg1, "string_argument", ...) YIELD res1, res2, ...; ``` -## General procedures +The optional parameter is used if you want to run the procedure on a +[subgraph](/advanced-algorithms/run-algorithms#run-procedures-on-subgraph). + +## General `mg` procedures Here is the list of procedures from the `mg` query module that can be used with all other query module files and their signatures: @@ -225,7 +158,7 @@ CALL mg.load("module1"); ``` Memgraph will reload `module1.py`, all its imported Python packages and it will conclude that there is a subdirectory `module1` which contains Python utility files for `module1.py` and it will reload them too. Note, that if `module1` directory contains some subdirectories, they will also get reloaded. By reloading, it is assumed clearing from the `sys` cache and deleting compiled code from the `__pycache__`. The repository which contains subdirectories can be organized in a different way too, so e.g. `module1/` and `module2/` folders can be placed directly in the `python/` folder. -## Procedures for `.py` query modules +## `mg` procedures for `.py` query modules Memgraph includes several built-in procedures for editing and inspecting Python module files. @@ -363,7 +296,12 @@ syntax: ```cypher CALL module.procedure([optional parameter], arg1, "string_argument", ...) YIELD res1, res2, ...; ``` -Every procedure has a first optional parameter and the rest of them depend on the procedure you are trying to call. The optional parameter must be result of the aggregation function [`project()`](/cypher-manual/functions#aggregation-functions). If such a parameter is provided, **all** operations will be executed on a projected graph. Otherwise, you will work on the whole graph stored inside Memgraph. +Every procedure has a first optional parameter and the rest of them depend on +the procedure you are trying to call. The optional parameter must be result of +the aggregation function +[`project()`](/cypher-manual/functions#aggregation-functions). If such a +parameter is provided, **all** operations will be executed on a projected graph. +Otherwise, you will work on the whole graph stored inside Memgraph. Each procedure returns zero or more records, where each record contains named fields. The `YIELD` clause is used to select fields you are interested in or all From 762046e85382714ac7f894b9c28e249ad8bbd27d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ante=20Pu=C5=A1i=C4=87?= Date: Mon, 4 Sep 2023 08:26:52 +0200 Subject: [PATCH 4/7] Simplify query module development steps --- pages/custom-query-modules.mdx | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/pages/custom-query-modules.mdx b/pages/custom-query-modules.mdx index abb9bcdaf..2ae88420c 100644 --- a/pages/custom-query-modules.mdx +++ b/pages/custom-query-modules.mdx @@ -55,10 +55,10 @@ then select a language you want to develop in. ### Download the MAGE image - Run the following command to download the MAGE image equipped for development inside of Docker containers: + Run this command to get the Memgraph & MAGE development Docker image: ```shell - docker run -p 7687:7687 memgraph/memgraph-mage:version-dev + docker run -p 7687:7687 memgraph/memgraph-mage:-dev ``` By running this command, you will get an image with the following tools @@ -159,9 +159,7 @@ then select a language you want to develop in. docker build -t memgraph-mage . ``` - ### Start Memgraph MAGE - - Run the following command: + Verify that the build is successful by starting the built image: ```shell docker run --rm -p 7687:7687 --name mage memgraph-mage @@ -173,7 +171,7 @@ then select a language you want to develop in. [C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or [Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). - ### Create the `dev` MAGE image + ### Create the `dev` image To create the `dev` MAGE image, run the following command: @@ -189,14 +187,9 @@ then select a language you want to develop in. docker run --rm -p 7687:7687 --name mage memgraph-mage:dev ``` - ### Rebuild everything - - If you make any changes to the module, you can stop the container and do a - rebuild. Additionally, if you don't want to rebuild everything, continue with the next step. - ### Copy the files to the container - Copy the files to the container named `mage`: + Copy the files to the `mage` container: ```shell docker cp . mage:/mage/ From 737c0b1ed9167a05f98ff21a1e3e49efe323b905 Mon Sep 17 00:00:00 2001 From: Vlasta Date: Thu, 7 Sep 2023 11:25:43 +0200 Subject: [PATCH 5/7] some-fixes --- pages/custom-query-modules.mdx | 165 +++++++------------- pages/custom-query-modules/contributing.mdx | 16 +- 2 files changed, 58 insertions(+), 123 deletions(-) diff --git a/pages/custom-query-modules.mdx b/pages/custom-query-modules.mdx index c0677f059..f83596cb8 100644 --- a/pages/custom-query-modules.mdx +++ b/pages/custom-query-modules.mdx @@ -44,40 +44,41 @@ select *+ New Module*. ![](/pages/custom-query-modules/memgraph_lab_query_modules.png) If you don't won't to develop from Memgraph Lab, you can: -- [install MAGE development environment from Docker Hub] -- [build MAGE using the `docker build` command] -- [build MAGE from source] +- install MAGE development environment from Docker Hub +- build MAGE using the `docker build` command +- build MAGE from source -then select a language you want to develop in. +Then, select a language you want to develop in. -## Install MAGE and import query modules +### Install MAGE and import query modules - + - + + The following steps will guide you through installing MAGE development + environment from Docker Hub and importing develooped query modules. + - ### Download the MAGE image + {

Download the MAGE image

} - Run this command to get the Memgraph & MAGE development Docker image: + Run the following command to get the Memgraph and MAGE development Docker image: ```shell docker run -p 7687:7687 memgraph/memgraph-mage:-dev ``` By running this command, you will get an image with the following tools - installed: `Python3`, `Rust`, `Clang`, `Make`, and `CMake`. This way, you can - copy files to the container, build them inside and load query modules into - Memgraph. + installed: `Python3`, `Rust`, `Clang`, `Make`, and `CMake`. - ### Develop a query + {

Develop a query

} Develop query modules in [Python](/custom-query-modules/python), [C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or [Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). - ### Start the MAGE container + {

Start the MAGE container

} Use the following command to start the MAGE container: @@ -91,15 +92,15 @@ then select a language you want to develop in. docker run --rm -p 7687:7687 --name mage memgraph-mage:1.4-dev ``` - ### Copy the files to the container + {

Copy the files to the container

} - Copy the files to the container named mage: + Copy the files to the container named `mage`: ```shell docker cp . mage:/mage/ ``` - ### Enter the container + {

Enter the container

} Position yourself inside the container as `root`: @@ -108,7 +109,7 @@ then select a language you want to develop in. ``` - If you performed the build locally, make sure to delete the cpp/build directory because you might be dealing with different architectures or problems with `CMakeCache.txt`. + If you performed the build locally, make sure to delete the `cpp/build` directory because you might be dealing with different architectures or problems with `CMakeCache.txt`. To delete it, run: @@ -118,7 +119,7 @@ then select a language you want to develop in. - ### Build MAGE + {

Build MAGE

} Build MAGE with the option to copy executables from `mage/dist` to `/usr/lib/memgraph/query_modules`: @@ -126,7 +127,7 @@ then select a language you want to develop in. python3 setup build -p /usr/lib/memgraph/query_modules/ ``` - ### Exit the container + {

Exit the container

} Everything should be ready to exit the container and load the query modules: @@ -139,7 +140,7 @@ then select a language you want to develop in. - Install MAGE with Docker build + Install MAGE with Docker Build. Create a Docker image directly from the [MAGE GitHub repository](https://github.com/memgraph/mage), instead of pulling it from the @@ -147,7 +148,7 @@ then select a language you want to develop in. - ### Download image or clone the repo + {

Download image or clone the repo

} Download a [specific release](https://github.com/memgraph/mage/releases) from the MAGE repository or clone the [repository](https://github.com/memgraph/mage) for the latest @@ -156,7 +157,7 @@ then select a language you want to develop in. ```shell git clone --recurse-submodules https://github.com/memgraph/mage.git && cd mage ``` - ### Build the MAGE tagged Docker image + {

Build the MAGE tagged Docker image

} Run the the following command: @@ -170,13 +171,13 @@ then select a language you want to develop in. docker run --rm -p 7687:7687 --name mage memgraph-mage ``` - ### Develop modules + {

Develop modules

} Develop query modules in [Python](/custom-query-modules/python), [C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or [Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). - ### Create the `dev` image + {

Create the `dev` image

} To create the `dev` MAGE image, run the following command: @@ -184,7 +185,7 @@ then select a language you want to develop in. docker build --target dev -t memgraph-mage:dev . ``` - ### Start the container + {

Start the container

} Start the container with the following command: @@ -192,7 +193,7 @@ then select a language you want to develop in. docker run --rm -p 7687:7687 --name mage memgraph-mage:dev ``` - ### Copy the files to the container + {

Copy the files to the container

} Copy the files to the `mage` container: @@ -200,7 +201,7 @@ then select a language you want to develop in. docker cp . mage:/mage/ ``` - ### Enter the container + {

Enter the container

} Position yourself inside the container as `root`: @@ -209,7 +210,7 @@ then select a language you want to develop in. ``` - If you performed the build locally, make sure to delete the cpp/build directory because you might be dealing with different architectures or problems with `CMakeCache.txt`. + If you performed the build locally, make sure to delete the `cpp/build` directory because you might be dealing with different architectures or problems with `CMakeCache.txt`. To delete it, run: @@ -219,7 +220,7 @@ then select a language you want to develop in. - ### Build MAGE + {

Build MAGE

} Build MAGE with the option to copy executables from `mage/dist` to `/usr/lib/memgraph/query_modules`: @@ -227,7 +228,7 @@ then select a language you want to develop in. python3 setup build -p /usr/lib/memgraph/query_modules/ ``` - ### Exit the container + {

Exit the container

} Everything should be ready to exit the container and load the query modules: @@ -244,7 +245,7 @@ then select a language you want to develop in. - ### Install dependencies + {

Install dependencies

} To build from source, you will need: - Python3 @@ -254,7 +255,7 @@ then select a language you want to develop in. - UUID - [Rust and Cargo](https://doc.rust-lang.org/cargo/getting-started/installation.html) - ### Set up the machine + {

Set up the machine

} Run the following commands: @@ -278,7 +279,7 @@ then select a language you want to develop in. && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* ``` - ### Download the MAGE source code + {

Download the MAGE source code

} Clone the [MAGE source code](https://github.com/memgraph/mage) from GitHub: @@ -286,8 +287,7 @@ then select a language you want to develop in. git clone --recurse-submodules https://github.com/memgraph/mage.git ``` - - ### Run the `setup` script + {

Run the `setup` script

} Run the following command: @@ -295,7 +295,7 @@ then select a language you want to develop in. python3 setup build -p /usr/lib/memgraph/query_modules ``` - The script will generate a `dist` directory with all the needed files. + The script will generate a `dist` directory with all the necessary files. It will also copy the contents of the newly created `dist` directory to `/usr/lib/memgraph/query_modules`. Memgraph loads algorithms and modules from @@ -305,7 +305,7 @@ then select a language you want to develop in. installation process. If you have any questions, contact us on **[Discord](https://discord.gg/memgraph).** - #### Set a different `query_modules` directory + {
Set a different `query_modules` directory
} The `setup` script can set your local `mage/dist` directory or any other directory as the default one by changing the value of the @@ -330,13 +330,13 @@ then select a language you want to develop in. Don't forget to copy the aforementioned files from `mage/dist` to ``. - ### Develop modules + {

Develop modules

} Develop query modules in [Python](/custom-query-modules/python), [C](/custom-query-modules/c), [C++](/custom-query-modules/cpp) or [Rust](https://github.com/memgraph/mage/tree/main/rust/rsmgp-example). - ### Start Memgraph + {

Start Memgraph

} Make sure your Memgraph instance is running: @@ -344,7 +344,7 @@ then select a language you want to develop in. sudo systemctl status memgraph.service ``` - ### Copy the query module + {

Copy the query module

} Copy your developed query module to `/usr/lib/memgraph/query_modules` or your directory if you changed the default location of query modules: @@ -361,17 +361,18 @@ then select a language you want to develop in. ## Querying -> Note that query modules are loaded into Memgraph on startup, so if your -> instance was already running, you would need to execute the following query -> inside one of the [querying -> platforms](https://docs.memgraph.com/memgraph/connect-to-memgraph) to load -> them: +Query modules are loaded into Memgraph on startup, so if your instance was +already running, you need to execute the following query inside one of the +querying platforms ([mgonsole](/getting-started/cli), [Memgraph +Lab](/data-visualization) or a [client library](/client-libraries)) to load +them: ```cypher CALL mg.load_all(); ``` -Lastly, run a query and test your module: +Run a query and test your module. All procedures are called using the `CALL` +clause: ```cypher MERGE (start:Node {id: 0})-[:RELATION]->(:Node {id: 1})-[:RELATION]->(:Node {id: 2}) @@ -379,10 +380,13 @@ CALL random_walk.get(start, 2) YIELD path RETURN path ``` -Since every query module is run as one transaction in Memgraph, the user can stop -the query module by [terminating the corresponding transaction](/memgraph/reference-guide/transactions). The user first needs -to find out the transaction ID using `SHOW TRANSACTIONS` command and then terminate it -using the `TERMINATE TRANSACTIONS ` command. +Since every query module is run as one transaction in Memgraph, the user can +stop the query module by [terminating the corresponding +transaction](/memgraph/reference-guide/transactions), but only if crucial parts +of the code, such as while and until loops, or similar points where the +procedure might become costly, are preceded with CheckMustAbort() function. The +user can then find out the transaction ID using `SHOW TRANSACTIONS` command and +terminate it using the `TERMINATE TRANSACTIONS ` command. ## Testing @@ -428,63 +432,4 @@ Lastly, run the e2e tests with python: ```shell python test_e2e -``` - -## I don't know what to do with this knowledge - -### Memgraph X cuGraph - - - -The development image with cuGraph support is not available yet. If you want to -develop cuGraph-powered query modules in Docker, do not hesitate to [contact -us](https://memgraph.com/community) about it. - - - -#### Building MAGE with NVIDIA cuGraph locally with Docker - what is this - -1. Download the MAGE source code from - [GitHub](https://github.com/memgraph/mage): - - ```shell - git clone https://github.com/memgraph/mage.git && cd mage - ``` - -2. Build the **MAGE × cuGraph**-tagged Docker image: - - ```shell - docker build -f Dockerfile.cugraph -t memgraph-mage . - ``` - -3. Start Memgraph-MAGE with the following command: - ```shell - docker run --rm --gpus all -p 7687:7687 -p 7444:7444 --name mage memgraph-mage - ``` - - - -You can now query Memgraph from querying platforms such as [Memgraph -Lab](https://memgraph.com/product/lab) or -[mgconsole](https://github.com/memgraph/mgconsole). - -If you made any changes while the Docker container was running, you need to stop -the container and rebuild the image. For a workaround, check [Development -process for MAGE with -Docker](/advanced-algorithms/install-mage). - - - -### Load custom query modules - -Once you start Memgraph, it will attempt to load query modules from all *.so and -*.py files from the default directories. MAGE modules are located at -`/usr/lib/memgraph/query_modules` by default, but Memgraph will also load any -modules located at `/var/lib/memgraph/internal_modules`. - -If the files were added while Memgraph was already running, they can be manually -loaded into Memgraph using the following query: - -``` -CALL mg.load_all(); ``` \ No newline at end of file diff --git a/pages/custom-query-modules/contributing.mdx b/pages/custom-query-modules/contributing.mdx index 67f0e0d24..19cb6c342 100644 --- a/pages/custom-query-modules/contributing.mdx +++ b/pages/custom-query-modules/contributing.mdx @@ -28,20 +28,10 @@ Conduct](https://github.com/memgraph/mage/blob/main/CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable behavior to tech@memgraph.com. -## Prerequisites +## Import developed query modules into Memgraph -- You have developed a query module by yourself and/or followed our tutorial for - [Python](/custom-query-modules/cpp) or - [C++](/custom-query-modules/cpp) - - - -The following steps depend on how you installed Memgraph and MAGE as we need -to import the modules. - - - -## Importing query modules into Memgraph +If you developed a query module you would like to become a part of the MAGE libraryThe following steps depend on how you installed Memgraph and MAGE as you need to +import the modules. From a4b7b15bda9c337163f17dc81c047ea47ad7621f Mon Sep 17 00:00:00 2001 From: Vlasta Date: Thu, 7 Sep 2023 13:50:01 +0200 Subject: [PATCH 6/7] contributing --- pages/custom-query-modules/contributing.mdx | 195 +------------------- 1 file changed, 8 insertions(+), 187 deletions(-) diff --git a/pages/custom-query-modules/contributing.mdx b/pages/custom-query-modules/contributing.mdx index 19cb6c342..596accfc7 100644 --- a/pages/custom-query-modules/contributing.mdx +++ b/pages/custom-query-modules/contributing.mdx @@ -21,195 +21,16 @@ and your contribution: - [**Memgraph**](https://github.com/memgraph/memgraph) - [**MAGE**](https://github.com/memgraph/mage) +Feel free to create an issue or open a pull request on our Github +repo to speed up the development. +Also, don't forget to throw us a star on GitHub. + +Your feedback is always welcome and valuable to us. Please don't hesitate to +post on [Discord](https://www.discord.gg/memgraph). + ## Code of Conduct Everyone participating in this project is governed by the [Code of Conduct](https://github.com/memgraph/mage/blob/main/CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code. Please report unacceptable -behavior to tech@memgraph.com. - -## Import developed query modules into Memgraph - -If you developed a query module you would like to become a part of the MAGE libraryThe following steps depend on how you installed Memgraph and MAGE as you need to -import the modules. - - - - - -**1.** Start the MAGE container with: - -```shell -docker run --rm -p 7687:7687 --name mage memgraph-mage:version-dev -``` - -Be sure to replace the `version` with the specific version, for example: - -```shell -docker run --rm -p 7687:7687 --name mage memgraph-mage:1.4-dev -``` - -**2.** Copy your local MAGE directory inside the container in order for Memgraph -to import the query modules: - -**a)** First, you need to copy the files to the container named `mage`: - -```shell -docker cp . mage:/mage/ -``` - -**b)** Then, you need to position yourself inside the container as root: - -```shell -docker exec -u root -it mage /bin/bash -``` - -> Note: If you performed the build locally, make sure to delete the `cpp/build` -> directory because you might be dealing with different architectures or -> problems with `CMakeCache.txt`. To delete it, run: -> -> `rm -rf cpp/build` - -**c)** After that, build MAGE with the option to copy executables from -`mage/dist` to `/usr/lib/memgraph/query_modules`: - -```shell -python3 setup build -p /usr/lib/memgraph/query_modules/ -``` - -**d)** Everything should be ready to exit the container and load the query -modules: - -``` -exit -``` - - - - -**1.** To create the `dev` **MAGE** image, run the following command: - -```shell -docker build --target dev -t memgraph-mage:dev . -``` - -**2.** Start the container with the following command: - -```shell -docker run --rm -p 7687:7687 --name mage memgraph-mage:dev -``` - - - -If you make any changes to the module, you can stop the container and do a -rebuild. Additionally, if you don't want to rebuild everything, you can: -1. Copy the changes to the container. -2. Perform a build inside the container. -3. Copy the executables to the `/usr/lib/memgraph/query_modules/` directory, - where Memgraph is looking for query modules. - -The process is the same as described in step **2** of the tab `Docker Hub`. - - - - - - -**1.** Make sure your Memgraph instance is running: - -``` -sudo systemctl status memgraph.service -``` - -**2.** Now, we need to copy our developed query module `random_walk.py` to -`/usr/lib/memgraph/query_modules`: - -```shell -python3 setup build -p /usr/lib/memgraph/query_modules -``` - - - - - -## Querying - -> Note that query modules are loaded into Memgraph on startup, so if your -> instance was already running, you would need to execute the following query -> inside one of the [querying -> platforms](/) to load -> them: - -```cypher -CALL mg.load_all(); -``` - -Lastly, run a query and test your module: - -```cypher -MERGE (start:Node {id: 0})-[:RELATION]->(:Node {id: 1})-[:RELATION]->(:Node {id: 2}) -CALL random_walk.get(start, 2) YIELD path -RETURN path -``` - -Since every query module is run as one transaction in Memgraph, the user can stop -the query module by [terminating the corresponding transaction](/fundamentals/transactions). The user first needs -to find out the transaction ID using `SHOW TRANSACTIONS` command and then terminate it -using the `TERMINATE TRANSACTIONS ` command. - -## Testing - -Test decoupled parts of your code that don't depend on Memgraph like you would -in any other setting. E2e (end to end) tests, on the other hand, depend on -internal Memgraph data structures, like nodes and edges. After running Memgraph, -we need to prepare the testing environment on the host machine. Position -yourself in the mage directory you cloned from GitHub. The expected folder -structure for each module is the following: - -```plaintext -mage -└── e2e - └── random_walk_test - └── test_base - ├── input.cyp - └── test.yml -``` - -`input.cyp` represents a Cypher script for entering the data into the database. -To simplify this tutorial, we'll leave the database empty. `test.yml` specifies -which test query should be run by the database and what should be the result or -exception. Create the files following the aforementioned directory structure. - -### input.cyp - -```cypher -MATCH (n) DETACH DELETE n; -``` - -### test.yml - -```shell -query: > - MATCH (start:Node {id: 0}) - CALL random_walk.get(start, 2) YIELD path - RETURN path - -output: [] -``` - -Lastly, run the e2e tests with python: - -```shell -python test_e2e -``` - -## Next steps - -Feel free to create an issue or open a pull request on our [Github -repo](https://github.com/memgraph/mage) to speed up the development.
-Also, don't forget to throw us a star on GitHub. :star: - - -## Feedback -Your feedback is always welcome and valuable to us. Please don't hesitate to -post on our [Discord](https://www.discord.gg/memgraph). +behavior to tech@memgraph.com. \ No newline at end of file From 382965567fcbeeda4cfcebe60dd1dc2fce428dc5 Mon Sep 17 00:00:00 2001 From: Vlasta Date: Fri, 8 Sep 2023 10:52:31 +0200 Subject: [PATCH 7/7] nvidida --- pages/custom-query-modules.mdx | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/pages/custom-query-modules.mdx b/pages/custom-query-modules.mdx index f83596cb8..a921d8225 100644 --- a/pages/custom-query-modules.mdx +++ b/pages/custom-query-modules.mdx @@ -52,7 +52,7 @@ Then, select a language you want to develop in. ### Install MAGE and import query modules - + @@ -355,7 +355,38 @@ Then, select a language you want to develop in. + + + + {

Download MAGE source code

} + Download the MAGE source code from GitHub: + + ```shell + git clone https://github.com/memgraph/mage.git && cd mage + ``` + + {

Build the image

} + + Build the *MAGE × cuGraph*-tagged Docker image: + + ```shell + docker build -f Dockerfile.cugraph -t memgraph-mage . + ``` + + {

Run the image

} + + Start Memgraph-MAGE with the following command: + + ```shell + docker run --rm --gpus all -p 7687:7687 -p 7444:7444 --name mage memgraph-mage + ``` + + + If you made any changes while the Docker container was running, you need to stop + the container and rebuild the image. + +