epaper-idf - epaper-idf-component

An ESP-IDF component and project for e-paper displays

---

License

Copyright © 2021 Jeremy Carter <jeremy@jeremycarter.ca>

This project is primarily released under the terms of the license contained in the file named LICENSE, which can be found in the top-level folder of this project. It also uses a bit of 3rd-party code, which is in turn primarily licensed under whichever licenses are referenced in each header (.h) or source (.c, .cpp, etc.) file, as per the original authors' preferences. A possibly non-exhaustive set of these other licenses is included in the top-level folder of this project in the files with names beginning with "LICENSE-".

---

[ This is a work in progress! Check back later for updates... ]

You can test this project at your own risk if you want, but it's not ready for release yet, so please don't expect all the listed features to be available or working properly yet. Some things may not even be implemented at all yet. Check back later for new developments and updates...

---

How to clone this project's git repository and submodules

# Clone the current stable version (well it's not really stable yet though):
git clone -b v0.1 --recursive https://gitlab.com/defcronyke/epaper-idf.git && \
cd epaper-idf

# (Optional) Or clone the current development version instead:
#git clone --recursive https://gitlab.com/defcronyke/epaper-idf.git && \
#cd epaper-idf

# Make sure you have "nodejs" and "npm" installed first, then install
# some 3rd-party Javascript dependencies for the web app portion:
./update-git-repos.sh

How to update this project if you've already cloned it before

# You can use this bash script for convenience:
./update-git-repos.sh

# (Optional) Or use the git commands instead:
#git pull; \
#git submodule update --init --recursive

---

Project Links

Epaper IDF Firmware Project

epaper-idf

• Homepage:
  https://defcronyke.gitlab.io/epaper-idf
  https://defcronyke.github.io/epaper-idf
  https://epaper-idf.defcronyke.com
• Git Repository:
  https://gitlab.com/defcronyke/epaper-idf
  https://github.com/defcronyke/epaper-idf

Dependencies (git submodules, you don't need to download these first)

epaper-idf-component

• https://gitlab.com/defcronyke/epaper-idf-component
• https://github.com/defcronyke/epaper-idf-component
• Web config demo:
  https://defcronyke.gitlab.io/epaper-idf-component
  https://defcronyke.github.io/epaper-idf-component
  https://epaper-idf-component.defcronyke.com
• Web config on your device:
  LAN: http://epaper
  Access Point: http://192.168.4.1
• Web config development:
  Dev Server: http://127.0.0.1:3000
  Production Server: http://127.0.0.1:8080

Adafruit-GFX-Component (3rd-party)

• https://gitlab.com/defcronyke/Adafruit-GFX-Component
• https://github.com/defcronyke/Adafruit-GFX-Component
• Forked from this original 3rd-party Arduino-based library to add ESP-IDF support:
  https://github.com/adafruit/Adafruit-GFX-Library

Details

1. An ESP32 ESP-IDF component for Espressif ESP32 microcontroller-based e-paper display firmware projects.
2. Has streamlined support for WiFi HTTPS over-the-air (OTA) firmware updates.
3. Aims to solve some of the usability problems that some of the other ESP32 e-paper display libraries have.
4. I currently only have one model of e-paper screen to test with: WaveShare 7.5" 640x384 b/w Gdew075T8
5. Only ESP32 ESP-IDF (Espressif's official FreeRTOS SDK) support is planned. This will not work for Arduino framework-based projects.

• I am making this as an attempt to eventually replace some existing solutions which are floating around online currently (at least replace them for my own usage), since everything I could find had too many outstanding issues, and I didn't like the way their code was organized personally.

---

Quick Method - Install the official pre-built version of the firmware

How to install unmodified firmware that's already built. This method is fastest and easiest

Prerequisites (Quick)

1. You need to have python and the pip utility installed. Install those however they're meant to be installed on your OS first.

2. Run the following command to install Espressif's esptool.py firmware flashing utility:

   # Install the python-based firmware flashing tool:
   pip install esptool
   
   # (Optional) If the above command doesn't work, try this one:
   #python -m pip install esptool

Instructions (Quick)

1. Plug your ESP32 device into your computer's USB port, then run one of the following commands to flash the pre-built firmware onto the device:

   # install the release version:
   bash <(curl -sL https://tinyurl.com/epaper-idf-flash)
   
   # (Optional) Or install the development version:
   #bash <(curl -sL https://tinyurl.com/epaper-idf-flash) master

2. Once the firmware is installed, the device will begin hosting a WiFi access point (AP) which you can connect to and access a configuration web page to set the WiFi SSID and password that the device will use to connect to your LAN WiFi. By default the access point will only allow one connection at a time for added safety, and it will shut down once the device is able to successfully connect to a WiFi network:

   Default WiFi Access Point Details:

   • SSID: wifi-net-15455
   • Password: T3oD cOneTioN! 143 2 psS@wRiDDd$i$^s
   • Config URL (work in progress...): http://192.168.4.1

---

ESP-IDF-based Method (recommended) - Local build and install custom firmware

The usual ESP-IDF way of building and installing your own custom version of the firmware. This method is recommended because OTA updates are supported and it's well-tested, but you might prefer a different method listed in another section

Prerequisites (ESP-IDF-based)

1. You need to have python and the pip utility installed. Install those however they're meant to be installed on your OS first.

2. You need to have nodejs and the npm utility installed. Install those however they're meant to be installed on your OS first. Sometimes npm is included with nodejs, and sometimes it's not.

3. Install the current stable version of Espressif's ESP32 IDF:
   https://docs.espressif.com/projects/esp-idf/en/stable/esp32/get-started/index.html

4. If on ESP32 IDF v4.2, their release is bugged, so you have to do the following additional steps to fix it:

   cd ~/esp/esp-idf && \
   git checkout remotes/origin/release/v4.2 && \
   git submodule update --init --recursive && \
   ./install.sh

To set up the project for OTA firmware updating ability, do this once

# Generate the DH param (this takes a really long time,
# sometimes dozens of minutes):
./gen-dhparam.sh

# Generate the auth certificates. Change the "esprog" argument below to
# your dev computer's DNS hostname, or leave it as-is and set your network's
# DNS config to recognise your dev computer at the hostname "esprog":
./gen-certs.sh esprog

# Copy the new certificates onto your development machine, wherever you'll
# be hosting the OTA firmware updating procedure from. The below command's
# argument is a valid scp destination path, and the destination should be
# somewhere containing an up-to-date cloned copy of this project's git
# repository:
./copy-certs.sh $USER@dev-machine:~/epaper-idf

To configure the firmware

# (Optional) Source the ESP-IDF each time you open a new terminal instance
# to make many of the helper scripts run a bit faster:
#. idf.env

# Open the firmware configuration (Kconfig) menu:
./menuconfig.sh

1. Make sure you configure the settings inside the menus named "[<>] Project ...".
2. You might want to change the device's "Local netif hostname" (default is "epaper") in the "Component config -> LWIP" menu as well.
3. At a minimum you'll have to select your e-paper device from the menu, otherwise compiling will give an error. You should definitely check the GPIO pin mappings while you're at it, since it's critical that you get those mappings correct if you don't want to break your e-paper display, and the defaults are likely not going to be correct for the way you wired up your devices.

To build the firmware

# Build the firmware. The version will be set as "v0.1.0" by default:
./build.sh

# (Optional) Specify the firmware short version when building. If
# the major or minor version changes, it breaks backwards-compatibility
# on purpose by changing the name of the EpaperIDF class:
#./build.sh v0.2

# (Optional) Specify the full firmware version number instead and it
# will use this exact version number for the built firmware:
#./build.sh v0.2.0

To install the firmware onto the ESP32 device

# Install the firmware you just built onto the device, and begin
# monitoring with a serial console. The firmware will be built first
# if necessary The version will be set as "v0.1.0" by default:
./flash.sh

# (Optional) Specify the firmware short version when building and
# installing. If the major or minor version changes, it breaks
# backwards-compatibility on purpose by changing the name of the
# EpaperIDF class:
#./flash.sh v0.2

# (Optional) Specify the full firmware version number instead and it
# will use this exact version number for the built firmware:
#./flash.sh v0.2.0

To view the ESP32 device's serial console

# View the serial console output to see what the device is doing, for
# monitoring and debugging:
./monitor.sh

---

OTA firmware updating instructions

1. Configure your LAN's DNS to point the hostname "esprog" at the IP address of your firmware dev computer, or change the "esprog" argument to your hostname when running the "./gen-certs.sh esprog" script, as mentioned in an earlier section.

2. Make sure you put the auth certificates in place on your dev computer first, using the "./copy-certs.sh" script as mentioned above. You should have copied them into a folder containing an up-to-date version of this project's git repository. Re-read the earlier instructions if the OTA updates aren't working properly for you.

3. Run the following script on your dev computer to build the new version of your firmware, and then begin hosting it for the device to do an OTA update:

   # Build the firmware if necessary, and begin serving it on an OTA
   # firmware updates HTTPS server. By default the version number will
   # start as "v0.1.0". To trigger the OTA update, the micro version
   # will be auto-incremented by +1 to the value in the file
   # version-micro.txt, for example "v0.1.0" -> "v0.1.1":
   # -----
   # KNOWN BUG: This always uses "v0.1" as the short version, so if you
   # don't want that, for now you'll have to include your desired short
   # version as an argument to this command, as in the first "(Optional)"
   # example below:
   ./serve.sh
   
   # (Optional) Specify the firmware short version number when building
   # and serving. If the major or minor version changes, it purposely
   # breaks backwards-compatibility by changing the name of the EpaperIDF
   # class. To trigger the OTA update, the micro version will be
   # auto-incremented by +1 to the value in the file version-micro.txt,
   # for example "v0.2.0" -> "v0.2.1":
   #./serve.sh v0.2
   
   # (Optional) Specify the full firmware version number when building
   # and serving. If the device is already running this version, it
   # won't do an OTA update during startup. Note that with this type
   # of invocation the micro version won't be auto-incremented, to
   # prevent an unnecessary OTA firmware update from happening:
   #./serve.sh v0.2.0

4. After the above script is finished building the firmware, it will start waiting for OTA update requests from the device. Reboot your ESP32 device to get it to connect and update itself with the new firmware version (or just wait for the deep sleep wakeup timer to fire if you're using deep sleep).

Whenever you want to load new firmware, run the "./serve.sh" script and wait for the firmware to finish building, then reboot the ESP32 device to load the new firmware onto it (or just wait for the deep sleep wakeup timer to fire if you're using deep sleep).

---

How to customize the firmware configuration website

The website for configuring the firmware can be easily customized as per your requirements. It's a webpack-based web app stored on a SPIFFS partition on the ESP32 device, and it gets compiled into the binary file build/www.bin. Webpack is used to ease development, package everything together nicely, and minify some of the files to save space on the ESP32 device.

1. The website's source files are in the components/epaper-idf-component/web directory, so those are the ones to modify, and they will be built into a production version of the site by webpack. To start developing, you can run the development server (a nodejs webpack-dev-server) by running the ./serve-web.sh script:

   # Serve the dev version:
   ./serve-web.sh

   With the development server running, you can use the following URL to preview the site as you're working on it:
   http://127.0.0.1:3000

2. (Optional) To run a development server which minifies the files first before serving them, you can use the ./serve-web-min.sh script if you want:

   # (Optional) Serve the minified dev version:
   ./serve-web-min.sh

3. (Optional) To run a production server (a nodejs http-server) which serves the final files that are output by webpack, you can use the ./serve-web-prod.sh script if you want:

   # (Optional) Serve the production version:
   ./serve-web-prod.sh

   Note that the production server's URL is different than the one for the development server. The port is different. You can use the following URL to view the production version of the site while you're running the production server:
   http://127.0.0.1:8080

4. (Optional) If you just want to build the website's source files with webpack, to make the production version of the site, you can use the ./build-web.sh script, but you don't need to do it really, since most of the other helper scripts will do it automatically anyway:

   # (Optional) Build the website:
   ./build-web.sh

The production version of the site gets built in the components/epaper-idf-component/public directory, which is where it gets deployed to the ESP32 device from, and it also gets copied into the components/epaper-idf-component/docs directory so a demo of it can be displayed on GitHub Pages. There is also a GitLab Pages demo of it available as well if you prefer.

---

Git Fork-based CI/CD Method (beta, no OTA) - Remote build and install custom firmware

How to fork this project with CI/CD support to remotely build custom firmware. OTA updates aren't supported yet

Prerequisites (Git Fork-based CI/CD)

1. You need to have python and the pip utility installed. Install those however they're meant to be installed on your OS first.

2. Run the following command to install Espressif's esptool.py firmware flashing utility:

   # Install the python-based firmware flashing tool:
   pip install esptool
   
   # (Optional) If the above command doesn't work, try this one:
   #python -m pip install esptool

Instructions (Git Fork-based CI/CD)

1. Fork these git repositories into your own GitLab account, by clicking each of the following links:

   • https://gitlab.com/defcronyke/epaper-idf

   • https://gitlab.com/defcronyke/epaper-idf-component

   • https://gitlab.com/defcronyke/Adafruit-GFX-Component

2. Clone the repos you forked in the previous step onto your machine:

   # Set you GitLab username and branches here:
   GITLAB_USER="defcronyke-fork"
   #GIT_REPO_VERSION_BRANCH="v0.1"  # (Optional) Set the git version branch.
   #GIT_REPO_BRANCH="master"        # (Optional) Set the git main branch.
   
   # Clone your GitLab repo fork:
   ./clone-git-fork.sh

3. Modify something, then commit and push the changes to your forked repo to trigger the remote building of the firmware:

   # (Optional) Uncomment the following lines to override
   # the default git main and version branches:
   #GIT_REPO_BRANCH="master"
   #GIT_REPO_VERSION_BRANCH="v0.1"
   #GIT_REPO_VERSION_BRANCH_ADAFRUIT="1.10"
   
   # (Optional) Uncomment the following lines to create
   # a tagged release version:
   #GIT_REPO_VERSION_TAG="v0.1.0"
   #GIT_REPO_VERSION_TAG_ADAFRUIT="1.10.0"
   
   # Commit the changes and push them, triggering a
   # CI/CD build:
   ./commit-git-repos.sh Some changes.

4. Install the epaper-idf firmware onto the ESP32 device:

   # Set your GitLab username, repo, and branch here:
   GITLAB_USER="defcronyke-fork"
   GITLAB_REPO="epaper-idf"
   GIT_REPO_BRANCH="v0.1"       # release version
   #GIT_REPO_BRANCH="master"    # (Optional) or development version
   #SERIAL_PORT="/dev/ttyUSB0"  # (Optional) port to flash
   #BAUD_RATE="115200"          # (Optional) speed to flash
   #FLASH_SIZE="4MB"            # (Optional) size to flash
   
   # Install your forked version of the firmware that you just
   # built. You may need to wait a few minutes for the build to
   # finish first.
   bash <(curl -sL https://gitlab.com/$GITLAB_USER/$GITLAB_REPO/-/raw/$GIT_REPO_BRANCH/flash-firmware-online.sh)

5. Once the firmware is installed, the device will begin hosting a WiFi access point (AP) which you can connect to and access a configuration web page to set the WiFi SSID and password that the device will use to connect to your LAN WiFi. By default the access point will only allow one connection at a time for added safety, and it will shut down once the device is able to successfully connect to a WiFi network:

   Default WiFi Access Point Details:

   • SSID: wifi-net-15455
   • Password: T3oD cOneTioN! 143 2 psS@wRiDDd$i$^s
   • Config URL (work in progress...): http://192.168.4.1

---

Latest Highlights

Some things listed in this section may not be fully implemented, tested, or working at all yet, but many of them are.

1. WiFi access point for hosting a WiFi connection settings web page:

   [*] Enable WiFi Access Point (AP)
   ***  ----- WiFi Access Point Settings -----  ***
   WiFi AP Startup (After Connection Retries)  --->
   (1)     WiFi AP Connection Retries Before Starting
   (wifi-net-15455) WiFi AP SSID
   (T3oD cOneTioN! 143 2 psS@wRiDDd$i$^s) WiFi AP Password
   (192.168.4.1) WiFi AP IP Address
   (1)     WiFi AP Channel
   (1)     WiFi Maximum connections AP
   ***  ----- End WiFi Access Point Settings -----  ***
   

   By default the access point only comes on if the device is unable to connect to its configured WiFi network after the configured number of connection retries. You can have the access point running all the time by changing the "WiFi Access Point Startup" option to "Always On", but note that this option will disable deep sleep if it was enabled.

2. CI/CD Pipeline added to the GitLab project, with an easy to use pre-built firmware flashing method:

   pip install esptool
   bash <(curl -sL https://tinyurl.com/epaper-idf-flash)

3. Lots of project-specific settings in the esp-idf Kconfig menu, so barely anything needs to be hard-coded in the C and C++ source code files:

   [<>] Project connection config --->
   [<>] Project display config --->
   [<>] Project OTA firmware config --->
   [<>] Project task config --->
   

4. Over The Air (OTA) HTTPS updates feature has been streamlined with auto-incrementing firmware micro version ( v0.1[.0] ), so it's really easy and painless to deploy new versions to the device over the air, using a few helper scripts:

   # Initial setup:
   ./gen-dhparam.sh
   ./gen-certs.sh [ota-server-hostname]
   ./copy-certs.sh [$USER@ota-server-hostname:]~/epaper-idf
   
   # Auto-increment micro version, build, then serve OTA:
   ./serve.sh [v0.1[.0]]

5. E-paper display device, its connections, and the desired program to run on it are selected in the esp-idf Kconfig menu:

   Select e-paper device (Gdew075T8) --->
   (device/Gdew075T8.h) e-paper device override
   *** ----- Display Settings ----- ***
   (0) Display rotation: 0 = 0°, 1 = 90° cw, 2 = 180° 3 = 270°
   (23) SPI GPIO for MOSI (MOSI or DIN)
   (18) SPI GPIO for Clock (CLK)
   ...
   *** ----- End Display Settings ----- ***
   

6. The minimal required set of header (.h) files for the selected e-paper device and main task are included automatically based on your Kconfig selections, and the header paths/filenames can be overridden in the Kconfig menu if you want to do something custom or weird:

   Select project main task (http-slideshow) --->
   (task/http-slideshow.h) Project main task override
   

7. Easy to add your own new programs (a.k.a. "main tasks") as options in the Kconfig menu:

   Select project main task (user)  --->
   (task/none.h) Project main task override (NEW)
   ***  ----- Task Settings -----  ***
   (-10) Deep sleep after refresh [delay_secs: ilong_min - -15 || deep_sleep_secs: 15 - ilong_max]
   ***  ----- End Task Settings -----  ***
   

8. You can choose to deep sleep between screen refreshes to save power if you want, or you can keep everything running and use a regular task delay instead:

   • deep sleep for 15 seconds:

     (15) Deep sleep seconds between screen refreshes [deep_sleep_secs: 15 - ilong_max]
     

   • (Optional) delay for 15 seconds instead, specify the number as negative:

     (-15) Deep sleep seconds between screen refreshes [delay_secs: ilong_min - -15]
     

9. The first example program "http-slideshow" (work in progress) connects to an HTTPS web server to fetch bitmap (.bmp) images, which will be displayed on the e-paper screen in sequence, as a slideshow:

   Select project main task (http-slideshow)  --->
   (task/http-slideshow.h) Project main task override
   ***  ----- Task Settings -----  ***
   (-10) Deep sleep after refresh [delay_secs: ilong_min - -15 || deep_sleep_secs: 15 - ilong_max]
   (https://defcronyke.gitlab.io/epaper-idf/http-slideshow/index.json) URL to a JSON object of paths to images (8-bit max .bmp)
   (defcronyke.gitlab.io) HTTP Host header value for above URL
   ***  ----- End Task Settings -----  ***
   

10. Adding a new e-paper device is made easier with the help of some C preprocessor macros. You can look at components/epaper-idf-component/include/device/Gdew075T8.h for an example of a real device, and notice that you can refer to every device as "class EpaperIDFDevice", which will be properly expanded to its full name under-the-hood.

11. The idea is that you'll add more devices and main tasks as per the included examples, and then they'll be selected and configured through the esp-idf Kconfig menu.

---

Acknowledgements

• Some of the code in this project is borrowed (and heavily modified/improved) from the OG e-paper library known as: ZinggJM/GxEPD
• Some of the code is loosely inspired by this other ESP IDF component project for e-paper displays: martinberlin/cale-idf

A big thanks to the authors of the above projects for releasing their code with permissive licensing, so I could derive some ideas from their existing work. Both of those projects have too many bugs though, and I wanted to make my own thing instead of trying to convince them to let me patch up their projects. Maybe at some point this will be a good alternative to those two projects, but until then, check out those ones because they are much more mature than this.