ESP RainMaker Admin CLI is a tool offered by Espressif Rainmaker for admin users to be able to perform mass manufacturing of nodes of ESP32-S2 and ESP32 based products. This tool will enable you to perform node id generation and certificate registration operations required for the manufacturing process.
Clone this project using:
git clone https://github.com/espressif/esp-rainmaker-admin-cli.git
Note: If you are using esp-idf, the python and virtual environment requirement would already be fulfilled (via install.sh), and you can just install the dependencies using
python -m pip install -r requirements.txtand move on to the next section which speaks about the workflow.
- Linux / MacOS / Windows (standard distributions)
To setup your build environment, please make sure you have the following installed on your host machine:
python(If not installed, please refer to https://www.python.org/)pip(If not present, please refer to https://pip.pypa.io/en/stable/)virtualenv(You can install using command -pip install virtualenv). This is not mandatory, but recommended so that rest of your python based utilities do not break.
The following python versions are supported:
- python 3.5.x
- python 3.6.x
- python 3.7.x
- python 3.8.x
Once python and pip are installed, set up the virtual environment by following the instructions here. Thereafter, please enter the directory where this tool is installed (using terminal) and execute the below to install the dependencies:
pip install -r requirements.txt
OR
python -m pip install -r requirements.txt
You need to perform the following steps to generate and register node credentials. To know more about the commands in detail, please refer the Usage section below.
- Set Server Configuration:
python rainmaker_admin_cli.py account serverconfig --endpoint <endpoint> - Login:
python rainmaker_admin_cli.py account login --email <email_id> - Generate Device Certificate(s):
python rainmaker_admin_cli.py certs devicecert generate [--videostream] [--no-pop] --count <count> - Register Generated Device Certificate(s):
python rainmaker_admin_cli.py certs devicecert register --inputfile <inputfile> - Check Device Certificate Registration Status:
python rainmaker_admin_cli.py certs devicecert getcertstatus --requestid <request_id>
To get help information for any RainMaker Admin CLI command or sub-command, you can use the -h option at various levels
Eg.
python rainmaker_admin_cli.py -h
python rainmaker_admin_cli.py account -h
python rainmaker_admin_cli.py account login -h
The Admin CLI commands are divided into 2 broad categories
- Account Operations
- Certificate Operations
You need to setup the account before you can move on to the Certificate Operations
You need to setup server configuration to get started. The endpoint would be your deployment's Base URL of the form https://xxxx/amazonaws.com/dev which you would have received on the super admin email configured during RainMaker deployment.
Usage:
python rainmaker_admin_cli.py account serverconfig --endpoint <endpoint>
You need to login to get started and use the subsequent APIs. The email id for login would be the super admin user email configured during RainMaker deployment. The password should have been already received on that email at the end of the backend deployment process.
Usage:
python rainmaker_admin_cli.py account login --email <emailid> [--password <password>]
Note: If password is not passed, it will be prompted for and not shown on screen for security reasons.
Note: Login configuration will be stored at location
~/.espressif/rainmaker/rainmaker_admin_config.json
To logout from the current session and clear stored credentials.
Usage:
python rainmaker_admin_cli.py account logout
Note: This will logout from the server and remove local session data.
You can now use the rest of the commands once you have logged in successfully.
You can perform the following operations for the device certificate.
generate- You can generate multiple device certificates at a time.register- You can register multiple generated device certificates.getcertstatus- Once you register the device certificates, you can check the device certificate registration status.
This will generate the private keys and certificates required by the RainMaker nodes to connect to your deployment. It will also set other information like the node ids, mqtt endpoint, etc.
Notes:
- This will also create the CA key and certificate that would be used for signing the device certificates. If you already have your own CA key and certificate, you can provide it explicitly.
- The created CA certificate and key will also be stored in a common folder named - 'ca_certificates' in the current working directory for reusing them for further device certificate generation. The key and the certificate will be stored in a sub folder under 'ca_certificates' named by the mqtt endpoint as prefix.
- If you want the Provisioning QR codes to be generated as well, please use the --prov option, and pass appropriate transport. Generally, the default is "ble" for all chips that support BLE (ESP32, ESP32-C3) and "softap" for the ones that do not (ESP32-S2). However, this primarily depends on what you have used in your firmware.
Usage:
python rainmaker_admin_cli.py certs devicecert generate [-h] [--outdir <outdir>] [--count <count>]
[--cacertfile <cacertfile>] [--cakeyfile <cakeyfile>]
[--prov <prov_type>] [--prov_prefix <prov_prefix>] [--fileid <fileid>]
[--cloud] [--local] [--inputfile <inputfile>] [--prefix_num <start> <length>]
[--videostream] [--no-pop] [--key_type <key_type>]
optional arguments:
-h, --help show this help message and exit
--outdir <outdir> Path to output directory. Files generated will be saved in <outdir>
If directory does not exist, it will be created
Default: current directory
--count <count> Number of Node Ids for generating certificates
Default: 0
--cacertfile <cacertfile>
Path to file containing CA Certificate
--cakeyfile <cakeyfile>
Path to file containing CA Private Key
--prov <prov_type> Provisioning type to generate QR code
(softap/ble) Default value: ble
--prov_prefix <prov_prefix>
Provisioning name (requires changes in firmware) Default is PROV
--fileid <fileid> File identifier
Used to identify file for each node uniquely (used as filename suffix)
Default: <node_id> (The node id's generated)
If provided, eg. `mac_addr`(MAC address),
must be part of ADDITIONAL_VALUES file (provided in config)
and must have <count> values in the file (for each node)
--cloud Use cloud-based node ID generation.
Default: local generation is used if not specified.
--local Use local node ID generation (default behavior).
This flag is redundant but kept for compatibility.
--inputfile <csvfile> This is the node_ids.csv file containing pre-generated node ids.
--prefix_num <start> <length>
These prefix numbers start (counter) and length (minimum length of digits as prefix) are added for each node specific output filenames as index. For example --prefix_num 1 4 will set file or folder name prefixes as node-0001-<node_id>.<file_extension if it is a file>. The prefixes follow order of 0001, 0002, 0003, etc as per the start (counter) value and the number of nodes for which to generate the device certificates (--count). The default value of the index is 1 (start) and 6 (length).
--videostream Require mqtt_cred_host to be present in the response. Will throw an error if not available.
--no-pop Generate QR code without pop field. When specified, the QR code payload will not include the 'pop' field.
--key_type <key_type> Cryptographic key type for device certificates. Options: 'rsa' (RSA 2048-bit, default) or 'ecdsa' (ECDSA P-256, faster and smaller).
For generating the node Ids locally without the rainmaker login:
python rainmaker_admin_cli.py certs devicecert generate --count 5 --prov ble --outdir test --local
For generating the node certificates by providing pre-generated node ids csv file:
python rainmaker_admin_cli.py certs devicecert generate --prov ble --outdir test --inputfile <node_ids.csv>
Note: In this command, the count and local arguments will be ignored, and the inputfile will take precedence, determining the number of Node IDs for which device certificates will be generated.
- The input file must be a CSV with a header row (field names as the first row).
- Node IDs will only be retrieved from rows under a single column named
node_id.
For generating device certificates with QR codes without the pop field:
python rainmaker_admin_cli.py certs devicecert generate --count 5 --prov ble --outdir test --no-pop
Note: When using
--no-pop, the generated QR codes will not include the pop field, which might be required for certain firmware implementations that don't use pop-based authentication.
For generating device certificates with ECDSA keys (faster, smaller certificates):
python rainmaker_admin_cli.py certs devicecert generate --count 100 --key_type ecdsa --outdir test
Note: ECDSA P-256 certificates are faster to generate and smaller in size compared to RSA, while providing equivalent security. RSA remains the default for compatibility.
For simplest use case, the usage is as given below. If you want to add some custom data or customise some other parameters, please refer the subsequent sections.
Note that it is better to first create a small set of certificates, say 5, so that you get an idea about how the tool works.
Example:
python rainmaker_admin_cli.py certs devicecert generate --count 5 --prov ble --outdir test
Sample result for 2 nodes is as below :
test
└── 2024-09-29
└── Mfg-000001
├── bin
│ ├── node-000001-T2uNDXPMS9nj9vpKjs2QG8.bin
│ └── node-000002-dRagJ6GBim2HE5ENQ5nbYG.bin
├── common
│ ├── ca.crt
│ ├── ca.key
│ ├── config_tmp.csv
│ ├── config.csv
│ ├── endpoint.txt
│ ├── mqtt_cred_host.txt # if --videostream option was given
│ ├── node_certs.csv
│ ├── node_ids.csv
│ ├── values_tmp.csv
│ └── values.csv
├── csv
│ ├── node-000001-T2uNDXPMS9nj9vpKjs2QG8.csv
│ └── node-000002-dRagJ6GBim2HE5ENQ5nbYG.csv
├── node_details
│ ├── node-000001-T2uNDXPMS9nj9vpKjs2QG8
│ │ ├── node.crt
│ │ ├── node.key
│ │ ├── qrcode.txt
│ │ └── random.txt
│ └── node-000002-dRagJ6GBim2HE5ENQ5nbYG
│ ├── node.crt
│ ├── node.key
│ ├── qrcode.txt
│ └── random.txt
└── qrcode
├── node-000001-T2uNDXPMS9nj9vpKjs2QG8.png
└── node-000002-dRagJ6GBim2HE5ENQ5nbYG.png
The output directory will have the following sub-directory structure:
<outdir>/<current_date>/Mfg-<no>- Sub-directory with the current date is created.
- A
Mfg-<no>sub-directory will be created where<no>is the batch number (which increments on each CLI run).
The output directory contains the following files:
-
bin/: For each device certificate, the corresponding NVS partition binaries are generated in this directory, which can be used to flash onto the device. File format:node-<index>-<node_id>.bin -
common/: This has some common files that are generated during the processca.crt: CA Certificate.ca.key: CA Key.endpoint.txt: MQTT Endpoint for this deployment.mqtt_cred_host.txt: Endpoint to obtain credentials for webrtc video streamingnode_certs.csv: CSV for all the Node Certificates in this batch to be registered to the cloud.node_ids.csv: CSV for all node ids generated in this batch.config.csv: The NVS configuration file as per the format defined here for the IDF Manufacturing Utility.values.csv: Master file with all the values for all the nodes as per the format defined here for the IDF Manufacturing Utility.
- The will be few '_tmp' files generated for values.csv and config.csv which will be used for internal purposes.
-
csv/:node-<index>-<node_id>.bin: For each device certificate, the corresponding csv file used as configuration to generate the binary.
-
keys/:keys-node-<index>-<node_id>.bin: For each device certificate, the corresponding encryption key (if encryption is enabled in config).the binary.
-
node_details/: All node details are stored in this directory. Following details for each node are stored innode_details/node-<index>-<node_id>directory:node.crt: Device Certificates.node.key: Private key for each device certificate.qrcode.txt: The QR code payload (used during provisioning, available only if --prov is given).random.txt: The random bytes information (used to generate device name and PoP, available only if --prov is given).
-
qrcode/: QR code images for all nodes are stored in this directory (used during provisioning, available only if --prov is given). File format:node-<index>-<node_id>.png
Adding Custom Data
There could often be a requirement to add some custom data to the nvs binaries generated. Such custom data can be added using the formats specified by the ESP IDF Manufacturing Utility. The config file and values file can be given as input by setting the ADDITIONAL_CONFIG and ADDITIONAL_VALUES fields in config/binary_config.ini. Please check out samples for such files at samples/extra_config.csv and samples/extra_values.csv
Note:
- When a valid values CSV file is specified in the
ADDITIONAL_VALUESfield, the--countargument is ignored, and the number of Node Ids for generating certificates is determined by the number of rows (excluding the header) in the provided CSV file.- If a
node_ids.csvfile is provided via the--inputfileargument, the row count from theADDITIONAL_VALUESCSV file is ignored, and the node count is determined by thenode_ids.csvfile instead.- The precedence for determining the node count for device certificate generation is as follows:
node_ids.csvfile passed via--inputfile- Values CSV file specified in the
ADDITIONAL_VALUESfield- The
--countargument (used only if neither of the above is provided).- When videostream feature is needed,
--videostreamswitch should be passed to instruct the tool to includemqtt_cred_hostin the factory partition.
Once the device certificates are generated, they also need to be registered with the cloud service using the register command.
Usage:
python rainmaker_admin_cli.py certs devicecert register [-h] --inputfile <csvfilename>
[--groupname <nodegroupname>] [--type <nodetype>]
[--model <nodemodel>] [--parent_groupname <parent_groupname>][--subtype <nodesubtype>]
[--tags <nodetags>]
[--force]
[--update_nodes]
[--node_policies <policies>]
[--skip_csv_validation]
optional arguments:
-h, --help show this help message and exit
--inputfile <csvfilename>
Name of file containing node ids and certs
--groupname <nodegroupname>
Name of the group to which node are to be added after successful registration
--type <nodetype> Node type
--model <nodemodel> Node model
--parent_groupname <parent_groupname>
Name of the parent group to which this newly created group will be a child group
--subtype <nodesubtype> Node SubType
--tags <nodetags> Comma separated strings of tags to be attached to the nodes.(eg: location:Pune,office:espressif)
--force Whether to ignore the error for duplicate node registration, also updates the existing certificates
--update_nodes Whether to skip registration of the device certificates and only add the type, model, subtype and tags to the nodes.
--update_nodes and --force If both are given, only the existing nodes will be updated with the new type, model, subtype and tags, Also the certificates will be updated.New nodes will be skipped.
--node_policies IoT access policies that need to be attached to the manufactured nodes, eg. videostream.
--node_policies option cannot be used together with --update_nodes. If both are provided, the command will fail.
--node_policies valid values: 'mqtt', 'videostream', or leave empty (default: mqtt). Multiple policies can be specified as comma-separated values (e.g., 'mqtt,videostream').
--skip_csv_validation Skip CSV validation (both certificate CN validation and column count validation). Use this option if you have want to bypass this check (NOT RECOMMENDED).For the example in device certificate generation section the node_certs_file file would be test/2020-11-29/Mfg-00001/common/node_certs.csv.
This command will give a request id in response, which can be used for monitoring the status.
Basic Tags can be added to nodes using the --tags option. Using --tags loc:Amsterdam will add this same tag to all the nodes in node_certs.csv.
Note that for adding tags, minimum rainmaker supported version is 1.1.27.
If different tag values are to be used for each node in the node_certs.csv file, use --tags loc:@city, where city should be a column in the node_certs.csv.
For example: node_certs.csv:
| node_id | certs | city |
|---|---|---|
| node1 | cert1 | Amsterdam |
| node2 | cert2 | Barcelona |
After passing --tags loc:@city,
- node1 will get the tag:
loc:Amsterdam. - node2 will get the tag:
loc:Barcelona.
To use CSV based tags, i.e
city:@locwhere loc is a column in the CSV, the minimum rainmaker supported version is 1.1.28.
The certificate registration process can take significant time. Once it is finished, the super admin user will get an email with the status. The same can also be checked using the getcertstatus command.
Usage:
python rainmaker_admin_cli.py certs devicecert getcertstatus --requestid XXXXXXX
Please check the output of the register command above to get the request id.
The steps here would generally not be required. However, if you want to explicitly create a new CA certificate and use it for signing the device certificates, you can use this command.
Usage:
python rainmaker_admin_cli.py certs cacert generate [--key_type <key_type>]
This will generate the CA key and certificate at following locations:
ca_certificates/<mqttendpoint>/ca.keyca_certificates/<mqttendpoint>/ca.crt
python rainmaker_admin_cli.py certs cacert generate --outdir <outdir> [--key_type <key_type>]
This will generate the CA key and certificate at following locations:
<outdir>/ca.key<outdir>/ca.crt
If there already exists CA a certificate and a key, then the existing ones are reused.
The --key_type option allows you to choose between 'rsa' (RSA 2048-bit, default) and 'ecdsa' (ECDSA P-256) for the CA certificate. Both can sign any type of device certificate.
These can be used for signing the device certificates by passing these via the --cacertfile and --cakeyfile options for rainmaker_admin_cli.py certs devicecert generate
To flash binary generated onto the device, you can use the following command:
esptool.py --port <port> write_flash <fctry_address> <outdir>/bin/<filename>.bin
Note: The
<fctry_address>is typically 0x340000. However, please check your partition table to find the appropriate address.The esptool.py would be available in your PATH only if you have esp-idf set up, else, please find it at
esp-idf/components/esptool_py/esptool/esptool.pyand use from there.
The download command allows you to fetch API responses from the RainMaker backend and save them to files. This is particularly useful for extracting paginated data (like node lists) and converting them to CSV format for analysis or reporting.
- Download API responses from any RainMaker API endpoint
- Extract paginated array data and combine it into a single CSV file
- Handle pagination automatically to fetch multiple pages
- Export data for offline analysis or reporting
python rainmaker_admin_cli.py download --api <api> [--out <folder>] [--csv_key <key>] [--csv_columns <columns>] [--query_params <params>] [--pages <num>]--api <api>(required): API endpoint path (e.g.,/admin/nodes)--out <folder>(optional): Output folder name where files will be saved. Default:downloads--csv_key <key>(optional): Key name in JSON response to extract as CSV. The value must be an array (e.g.,node_info). If not provided, only the API response will be saved (useful for understanding the API structure).--csv_columns <columns>(optional): Comma-separated list of column names for CSV. Supports dot notation for nested keys (e.g.,connectivity.timestamp). If provided, CSV will only contain these columns in the specified order.--query_params <params>(optional): Query parameters for API request (e.g.,node_list=true&status=online)--pages <num>(optional): Number of pages to query. Default: 1. Use0to query all pages until end.
When the command executes, it creates a timestamped subdirectory in the output folder to organize files from each run:
<output_folder>/
└── <timestamp>/
├── api_request.txt
├── api_response.txt
└── list.csv (only created if --csv_key is provided)
The timestamp format is YYYYMMDD_HHMMSS (e.g., 20251113_105742), ensuring each run is isolated in its own directory.
File descriptions:
api_request.txt: Summary of the API request including URL, query parameters, CSV settings, and pagination statisticsapi_response.txt: Contains the first page API response (without the csv_key if--csv_keyis provided)list.csv: Contains the extracted CSV data (only created if--csv_keyis provided)
The download command automatically handles pagination:
- Default behavior: Fetches only the first page (
--pages 1or omitted) - Multiple pages: Use
--pages <num>to fetch a specific number of pages - All pages: Use
--pages 0to fetch all pages until the end - Pagination detection: Stops when
next_idis absent,null, or"null"in the response - Query params: User-provided
--query_paramsare included in all page requests - CSV accumulation: When using
--csv_key, items from all fetched pages are combined into a single CSV file
Basic download (first page only, to default folder):
python rainmaker_admin_cli.py download --api "/admin/nodes"Basic download with custom output folder:
python rainmaker_admin_cli.py download --api "/admin/nodes" --out "my_data"Download with CSV extraction:
python rainmaker_admin_cli.py download --api "/admin/nodes" --csv_key "node_info"Download with custom CSV columns:
python rainmaker_admin_cli.py download --api "/admin/nodes" --csv_key "node_info" --csv_columns "node_id,node_status,registration_timestamp"Download with nested field extraction (dot notation):
python rainmaker_admin_cli.py download --api "/admin/nodes" --csv_key "node_info" --csv_columns "node_id,status.connectivity.connected,status.connectivity.timestamp"Download with query parameters:
python rainmaker_admin_cli.py download --api "/admin/nodes" --csv_key "node_info" --query_params "status=online&node_list=true"Download multiple pages:
python rainmaker_admin_cli.py download --api "/admin/nodes" --csv_key "node_info" --pages 5Download all pages:
python rainmaker_admin_cli.py download --api "/admin/nodes" --csv_key "node_info" --pages 0Complete example with all options:
python rainmaker_admin_cli.py download --api "/admin/nodes" --out "nodes_export" --csv_key "node_info" --csv_columns "node_id,node_status,registration_timestamp,status.connectivity.timestamp" --query_params "status=online" --pages 0- Authentication: You must be logged in to use this command (see Login)
- Understanding API structure: Run the command without
--csv_keyfirst to see the API response structure and identify which key contains the data you want to extract - CSV key exclusion: When
--csv_keyis provided, the csv_key is excluded fromapi_response.txtto avoid storing large paginated arrays - Dot notation: Use dot notation (e.g.,
status.connectivity.timestamp) to access nested JSON fields in--csv_columns - Array handling: Arrays and nested objects in CSV are converted to JSON string format
- Column ordering: If
--csv_columnsis provided, columns appear in the exact order specified. Otherwise,node_idis placed first (if present), followed by other columns alphabetically - Directory organization: Each run creates a timestamped subdirectory with all related files grouped together
- Please get in touch with your ESP RainMaker contact in case of any issues or send an email to esp-rainmaker-support@espressif.com