Porting Guide for Xively C Client
Copyright (c) 2003-2018, Xively
Table of Contents
- Target Audience
- C Client Overview
- Supported Platforms
- Board Support Package (BSP)
- Porting the C Client
- Further Reading
This document is intended for Embedded Developers who have access to the source code of the Xively C Client and want to run it on their custom platform.
Details of the Xively C Client build steps, build configuration flags, and the location of the functions needed to port the library to new platforms are included in this document.
For information about the Xively Python client, please visit its public git repository: xiPy.
C Client Overview
The Xively C Client is designed for low resources platforms. The single threaded implementation is non-blocking, which allows it to operate on NoOs platforms. The implementation also supports seamless concurrent processing of outgoing and incoming messages.
An event system is included in the client which can be used by your client application as well. The client's event system is a platform independant implementation that doesn't require porting to any platform-specific timers or tasks.
The client provides this functionality while remaining small and efficient enough to fit into low-resource devices, with minimal footprint and processing power requirements.
The codebase is built against C99 standards.
For more information please see the Xively C Client User Guide:
The Xively Client has been deployed on many different devices shipping in the IoT space running the following platform stacks. The following is a catalog of our tested ports:
- POSIX systems (OSX, Linux)
- Microchip TCP SDK
- Texas Instruments CC3200
- Texas Instruments CC3220SF
- STM32 NUCLEO-F429ZI Ethernet
- STM32F4 CLOUD-JAM RushUp WiFi
- STM32 NUCLEO-F401RE (with IDW01M1 WiFi)
- STM32 NUCLEO-L476RG (with IDW01M1 WiFi)
- STM32 NUCLEO-L053R8 (with IDW01M1 WiFi)
Porting the Xively C Client to new platforms is accelerated by the Board Support Package (BSP).
Board Support Package (BSP)
The Board Support Package (BSP) is the well-defined set of functions that the Xively Client invokes to interact with a platform's specific networking, file IO, TLS, memory management, random number generator, time and firmware management SDKs.
We've organized the sources of the Xivey Client so that your BSP implementation resides in a few files of the directory
src/bsp. You should focus your attention here when porting the Xively Client to your device SDK, safely ignoring the MQTT codec or the non-blocking / asynchronous engine that resides in the rest of the main library source.
All of the BSP function declarations can be found under the
directory. Doxygen documentation for these functions can be found in the
BSP Functions are broken down by logical subsystems as follows:
- BSP FWU: Firmware Update Inegration (
- BSP IO FS: File System Integration for Secure File Transfer (SFT) (
- BSP IO NET: Networking Stack Integration (
- BSP TLS: Transport Layer Security Integration (
- BSP MEM: Heap Memory Management (
- BSP RNG: Random Number Generator (
- BSP TIME: Time Function (
Reference function implementations are separated into two directories:
The platform directory contains reference BSP implementations for networking, file IO, memory management, random number generator, time and firmware management. The tls directory contains reference implementations for wolfSSL and mbedTLS libraries which supply secure TLS v1.2 connections over TCP/IP.
Numerous platform implementations are provided for your reference. These are available
The POSIX implementation is probably the most familiar to most engineers. However, some of the other reference BSP implementations might be closer to the platform that you're working on, if you're working in the embedded device space.
If your platform is not supported by one of these reference implementations then we recommended that you:
- copy an implementation into a new directory
BSP TLS reference implementations for wolfSSL and mbedTLS can be found in a
If neither wolfSSL nor mbedTLS fits your target platform the build system can be configured to use other BSP TLS implementations.
Before selecting another TLS solution other than those mentioned above, be sure that it supports:
- asynchronous networking
- accepting Root CA certs to verify Xively's Server certificate
- Online Certificate Status Protocol (OCSP) or an actively maintained Certificate Revocation List
- The added functionality of OCSP Stapling is prefered.
To create a new BSP implementation for TLS:
- implement all of the BSP TLS API functions found in
include/bsp/xi_bsp_tls.h. It is advised to reference at least one of the wolfSSL or mbedTLS implementations throughout this process to guide your development.
- put this implementation into directory
- copy the file
make/mt-config/mt-tls-[NEW_TLS_LIBRARY_NAME].mkand set the path variables inside according to the new TLS library's internal directory structure
Customizing Both: Platform and TLS BSPs
make without parameters will silently default the configuration to
make XI_BSP_PLATFORM=posix XI_BSP_TLS=wolfssl.
This is fine for building on OSX or Linux/Unix using wolfSSL for a TLS library. However, for cross-compilation you will most likely need to define both parameters explicitly. For instance:
`make XI_BSP_PLATFORM=cc3200 XI_BSP_TLS=microSSL`
Note: microSSL is a fictional TLS library we've conjured-up as a quick custom example.
This command assumes directory
src/bsp/platform/cc3200 containing networking, memory, random and time implementations as well as directory
src/bsp/tls/microSSL covering the TLS calls.
Porting the C Client
Building a Xively C Client consists of building the Xively C Client library, the TLS library and linking them to the actual application. Obviously these steps need to be tailored to the target platform.
Suggested First Step - A Native Build
We recommend building the Xively Client on OSX or Linux/Unix before attempting to cross compile the sources to another target platform. This process could serve as an introduction to the basic architecture, tools and building steps of the Xively Client source. This basic knowledge will be handy later when doing a full cross-compiled port of the software to an embedded device.
Build System: make
The Xively C Client uses make to build for OSX or Linux/Unix. The main makefile is in the root directory of the repository.
Further build configurations can be found under directory
make/ which contains numerous make target (mt) files which are included by the main makefile. Here
make/mt-os/mt-os.mk files should be your main focus. These are always included by the main makefile, and further include optional mt files based on the CONFIG and TARGET flags you specify when building.
For instance, the
make/mt-os/mt-cc3200.mk file is included by
make/mt-os/mt-os.mk when building for the TI CC3200.
More information about CONFIG and TARGET flags can be found in the respective sections below.
Additionally you can run with
make MAKE_DEBUG=1 defined and our makefiles will log many of the source, include and flag variables that it's compiling so you can see where certain flags reside.
Build System: IDE Builds
Although the make build system is suitable for OSX or Linux/Unix builds, some MCU platforms may not support building via make. Often these devices supply their own Integrated Development Environment (IDE). For instance, the IAR toolchain uses their own graphical IDE.
The key to building the Xively Client with an IDE is to ensure that you have the correct Preprocessor Definitions defined and to import the correct set of files.
Here's a a suggestion on how to determine which flags and files you should import:
Executing a makefile build on OSX or Linux/Unix with your desired config flags (see below) and then examining the make output is highly recommended.
Additionally, by calling make with attribute 'MD=' you can reveal compiler and linker parameters, showing you which Preprocessor Definitions need to be migrated to your IDE environment.
Importing these definitions in an IDE varies considerably by IDE. Some IDEs let you define these in a source or configuration file. Others require you to enter them as fields in a project configuration window.
A vast majority of the sources in the
src/libxively directory Xively Client Library are applicable to all platforms, with the platform sepecific files residing in the
However some source files might not be needed depending on your configuration choices. Once again, like the preprocessor flags, we recommend running a make build with your desired configuration to determine which source files need to be added to your IDE builds, and which includes paths should be set.
First Build on OSX or Linux/Unix
Firing a bare
make should be enough for an OSX build. It defaults to posix platform and to wolfSSL for a TLS solution. It also includes the download (from github repo) and build of the wolfSSL library.
Alternatively one can use mbedTLS instead of wolfSSL if preferred:
Occasionally cleaning the build directories are necessary. This is done with either of the following commands:
make clean make clean_all
Building the Examples
From the base directory of the repository:
cd examples make
Here if the library was previously built with the mbedTLS setting then the XI_BSP_TLS variable is necessary. Otherwise you will get link errors.
- autotools: autoconf, automake, libtool are needed by wolfSSL
- cmake is needed by the integration test framework cmocka
On OSX these are available through
- brew install autoconf - brew install automake - brew install libtool - brew install cmake
On Ubuntu these are available through
- sudo apt-get update - sudo apt-get install autoconf - sudo apt-get install autotools-dev - sudo apt-get install libtool - sudo apt-get install cmake
Xively C Client Build Flags
The make system uses dash '-' separated string to define the TARGET and CONFIGuration for a build.
TARGET defines the build environment for specific platforms and toolchains.
CONFIG flags specify selections from a list of Xively Client modules and features to be included in the Xively C Client library.
Default values will be chosen for you if you do not set the TARGET or CONFIG makefile variables.
The TARGET and CONFIG flags are enumerated below, and can be mixed and matched in any order. Each flag is separated by the - (minus sign / dash) character.
A typical TARGET flag looks like this:
- [ osx | linux | arm-linux ] - selects the target platform
Output Library Type
- [ static | dynamic ] - selects the output library type
- [ debug | release ] - selects the build type
A typical CONFIG flag:
Xively Client Feature flags
- control_topic - turing on this feature makes the Client establish a control topic channel to the Xively Services. This channel is the transport layer for background Xively Service tasks like Secure File Transfer and Firmware Update. - secure_file_transfer - enables the Secure File Transfer and Firmware Update features. SFT keeps certain files up-to-date by defining a file set in the API function 'xi_set_updateable_files'. The control_topic flag must be turned on as well. - senml - turns on SENML JSON serialization support for timeseries data. To maintain a small footprint size we recommend turning this off (by removing senml flag from CONFIG). - threading - POSIX only. This causes pub, sub, and connection callbacks to be called on separate thread. Not having this flag set, application callbacks are called on the sole main thread of the Xively C Client.
File System flags
- posix_fs - POSIX implementation of File System calls - memory_fs - an in-memory File System implementation - dummy_fs - empty implementation of File System calls - expose_fs - adds a new Xively C Client API function which allows external definition of File System calls. This will be part of our BSP system in the future.
- memory_limiter - turns on memory limiting and monitoring functionality. The purpose of this configuration is to aid development processes by simulating low-memory environment behavior. The memory monitor part is useful for memory consumption tracking and to hunt down memory leaks. When a leak occurs a stack trace of the initial allocation will be logged. - mqtt_localhost - the Client will connect to localhost MQTT server. The purpose of this configuration option is to help development processes. - no_certverify - lowers security by disabling TLS certificate verification - tls_bsp - the Client will use third party TLS 1.2 implementations to encrypt data before it's sent over network sockets. The lack of this flag will skip the TLS handshaking. This be may helpful in connecting to mosquitto to test, but may not be supported on some secure services and is not recommend for production deployment. The Xively Gateway will not accept connections without TLS.
Platform selector flags
- posix_platform - selects implementation for non-BSP time and non-BSP memory solutions - wmsdk_platform - selects implementation for non-BSP time and non-BSP memory solutions
The Platform Selector configurations will eventually be sunset. Currently these configure the build system to include Critical Section implementations for invoking callbacks on new threads.
We suggest defining
posix_platform and omit
threading from your CONFIG options when building for custom platforms.
threading is currently ommitted by default.
For more information about thread safe callback support please see the Xively C Client User Guide:
Example make Command
By executing a simple 'make' under the base directory of the repository should be sufficient on OSX or Linux/Unix. This will result in build configuration with the following default flags:
- CONFIG: posix_fs-posix_platform-tls_bsp-senml-control_topic-memory_limiter-secure_file_transfer - TARGET: osx-static-release
The result will be a release version Xively C Client static library with debug outputs, secure TLS connection, POSIX networking and file system, artificial memory limits and memory guards turned on, SENML support for timeseries formatting, and SFT features enabled.
The development version flags without SENML and memory limits may look like this:
- CONFIG=posix_io-posix_fs-posix_platform-tls_bsp-control_topic-secure_file_transfer - TARGET=osx-static-debug
For CI configurations please look at the file .travis.yml.
Example Xively Client Applications Linked with the Library
Application binaries and source can be found under the directory
These examples use the Xively C Client Library for connecting to Xively Servers, subscribing to topics, publishing data, and receiving data. They can be built on POSIX by running
make in the
The examples are command line applications that link against the Xively C Client library. They require a Xively-specific account-id, username, password and optional topic name to subscribe or publish on.
Porting the Xively C Client to Your Platform
As a rule of thumb if you are stuck or confused, then please use existing platform config files like:
make/mt-os/mt-linux.mk files as examples.
The following is written assuming a new ficitious platform named the np2000.
The Goal: Cross Compilation with the make Build System
The ideal goal to cross-compile the Xively C Client on OSX or Linux/Unix to your platform. For the sake of this tutorial, let's pretend that your platform is a new fictitious platform: The NP2000. Your platform sounds awesome.
Our goal is construct the following command to build the Xively Client for the NP2000:
To mop up generated files type
make PRESET=np2000 clean
To make this possible, the following steps have to be taken.
create a new file
include the common mt file
add the lines
XI ?= $(XI_BINDIR)/libxively.a XI_COMPILER_FLAGS += -DNO_WRITEV // optional. this turns off a wolfssl feature which is not available on micro platforms
define CC and AR and point them to your compiler and archiver executables, e.g.:
CC = $(XI_CC3200_PATH_CCS_TOOLS)/compiler/ti-cgt-arm_15.12.1.LTS/bin/armcl AR = $(XI_CC3200_PATH_CCS_TOOLS)/compiler/ti-cgt-arm_15.12.1.LTS/bin/armar
add compiler flags by appending them to variable XI_COMPILER_FLAGS. We don't know what your flags would be, but perhaps they might look like this:
XI_COMPILER_FLAGS += -I$(XI_CC3200_PATH_SDK)/simplelink/include XI_COMPILER_FLAGS += -mv7M4
add archiver flags by appending them to the variable XI_ARFLAGS, like this:
XI_ARFLAGS := r $(XI)
make/mt-config/mt-presets.mkby adding the following details for your new platform:
define a Xively Client feature and target configurations which will be used in this same file afterwards
CONFIG_NP2000_MIN = memory_fs-tls_bsp TARGET_NP2000_REL = -static-release
define make system variables for PRESET np2000
else ifeq ($(PRESET), np2000) CONFIG = $(CONFIG_NP2000_MIN) TARGET = $(TARGET_NP2000_REL) XI_BSP_PLATFORM = np2000 XI_TARGET_PLATFORM = np2000
make/mt-os/mt-os.mkto check the TARGET make parameter for 'np2000' and include the
make/mt-os/mt-np2000.mkconfig file when the np2000 TARGET is found:
XI_CONST_PLATFORM_NP2000 := np2000 ifneq (,$(findstring $(XI_CONST_PLATFORM_NP2000),$(TARGET))) XI_CONST_PLATFORM_CURRENT := $(XI_CONST_PLATFORM_NP2000) endif
provide BSP implementations for all modules.
- Create the following source files:
- in these sourcefiles, define the functions declared in these corresponding Xively C Client BSP headers:
- firmware update notifications (
- file storage (
- networking (
- memory (
- time (
- random (
- firmware update notifications (
Hint: to attain a simple successful build just create the files and implement all the BSP API functions with an empty body. You may use files under
src/bsp/platform/dummyas an example empty starting point. While this won`t execute properly, it should at least build, link and run.
- Create the following source files:
select TLS implementation
the default selection is wolfssl. If wolfssl fits your needs then there's nothing to do here.
to select a different TLS lib add
to the make commandline like this:
make PRESET=np2000 XI_BSP_TLS=myTLSlibrary
for wolfssl and mbedtls the BSP TLS implementations are available in files:
if you chose a third one: myTLSlibrary then you have to write your own implementation
- create a file
src/bsp/tls/myTLSlibrary/xi_bsp_tls_myTLSlibrary.cand implement all functions declared in
- as samples you can follow the two existing implementations: wolfssl and mbedtls
- create a file
create a file
make/mt-config/mt-tls-myTLSlibrary.mkand fill in with content similar to
make/mt-config/mt-tls-mbedtls.mk. This is where you would defined the include directory, the static library directory, the static library file to link against and the config flags of the custom TLS library.
Using a Third Party IDE (non-make)
In the case that the make build system is unavailable for your platform you can migrate the Xively C Client build environment to your platform's own Integrated Development Environment (IDE).
While we cannot completely predict how this process would work for every IDE and toolchain, here are some suggested steps to follow when working through this process:
import all of the Xively C Client source files in the
src/libxivelydirectory into your project workspace. This is where all of the Xively Client's platform independent code resides.
- Some of these files might not be required depending on your desired library configuration. You could prune some files later by comparing the fileset against a mac / linux build log.
import all of the source files in one of the subdirectories of
src/bsp/platform. The Xively C Client BSPs contain all of the missing hooks which tie the platform independent code to a particular device. Complete implementations should include a source file for each of the BSP subsystems (networking, file IO, memory, time, firmware, etc).
- NOTE: The Xively C Client contains reference BSP implementations for POSIX, TI CC3200, and some ST32F4 Nucleo platforms. We also have partial implementations for specific networking APIs.
Modules from these 'incomplete' BSP implementations could be used as substitutes. For instance, on devices that mirror POSIX completely except for networking, like Simplelink, the networking module from
src/bsp/platform/posixcould be ovewritten with the one from
- NOTE: The Xively C Client contains reference BSP implementations for POSIX, TI CC3200, and some ST32F4 Nucleo platforms. We also have partial implementations for specific networking APIs. Modules from these 'incomplete' BSP implementations could be used as substitutes. For instance, on devices that mirror POSIX completely except for networking, like Simplelink, the networking module from
import one of the BSP TLS implementations in
src/bsp/tls. Currently we provide two different TLS BSP implementations: WolfSSL or mbedTLS.
alter the include path for the toolchain as follows:
- add all of the directories in
src/libxivelyto your include path
- add the
includedirectory to your include path
- add the
include/bspdirectory to your include path
- add any corresonding preprocessor defintions to toggle on/off Xively client features.
- add all of the directories in
Using the CONFIG flags in file
make/mt-config/mt-config.mk as a guide, the compiler flags used in the "Preceding Step" can be looked up and fed to the platform specific build system as well. Another option is to echo the makefile build system variable XI_CONFIG_FLAGS during building on OSX to see which flags are set.
For more informationa about the Xively Client, check these other documents in the github repository:
Contains general information about the file structure of the sources, how to build on OSX and Linux, a general overview of Security.
Contains an in-depth description of the Client Design and IoT Features, including MQTT logic, Event System, Back-Off Logic, Platform Security Requirements, etc.
Contains the function specifications for communicating with the Xively C Client from our application. Functionality such as Connect, Subscribe, and Publish are outlined here.
Contains the declarations and documentations of the abstracted Board Support Package functions that you will need to implement to couple the Xively C Client to your hardware platform.