Skip to content
/ buffy Public

An embeddable, MIT-licensed, C-language, zero-dependency memory buffer class inspired by libevent's evbuffer. It consists of just a few files that can be dropped into your own project as-is.

License

Notifications You must be signed in to change notification settings

ckerr/buffy

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

54 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

libbuffy

libbuffy is an embeddable, MIT-licensed, C-language, zero-dependency memory buffer class inspired by libevent's evbuffer. It consists of just a few files that can be dropped into your own project as-is.

Some common uses:

  • A pipeline between data producers and consumers
  • Queueing data that's been received or about to be sent over the network
  • Building strings or data buffers without worrying about memory management

Buffy is designed to be as efficient as possible despite being a general-purpose tool. It avoids unnecessary memory allocations and copies when possible.

Building libbuffy

$ git clone https://github.com/ckerr/buffy.git
$ cd buffy
$ git submodule update --init
$ mkdir build
$ cd build
$ cmake ..
$ make

Concepts: Pages, Content, and Space

bfy buffers are implemented using an array of separate pages, where a page is a chunk of contiguous memory. Page memory that's been written into via bfy_buffer_add*() is the content, while the as-yet-unused memory at the end of the last page is the buffer's free space. These terms are reflected in the API, e.g. bfy_buffer_get_content_len().

Most of the time, no special steps are needed for efficient use. bfy will use space available when possible and recycle leftover memory from content that's already been consumed by bfy_buffer_remove*() or bfy_buffer_drain*().

Sometimes, though, you may have data with special constraints: it may be read-only, or it may be memory-managed by someone else and bfy shouldn't free it, or you may want to transfer it from a different buffer. You can add these with the functions bfy_buffer_add_reference(), bfy_buffer_add_readonly(), or bfy_buffer_add_buffer(). These all create new pages in the buffer's array of pages so pre-existing content can be embedded into the buffer without the overhead of cloning it.

API

Life Cycle

Buffers can be instantiated on the heap:

bfy_buffer* bfy_buffer_new(void);
void bfy_buffer_free(bfy_buffer*);

If you're building bfy inside your own project such that bfy's ABI is not a concern, you can also instantiate it on the stack or aggregate it in a struct to avoid another malloc/free pair for the bfy_buffer struct itself:

bfy_buffer bfy_buffer_init(void);
void bfy_buffer_destruct(bfy_buffer*);

Adding Data

size_t bfy_buffer_add(bfy_buffer* buf, void const* addme, size_t len);
size_t bfy_buffer_add_ch(bfy_buffer* buf, char addme);
size_t bfy_buffer_add_printf(bfy_buffer* buf, char const* fmt, ...);
size_t bfy_buffer_add_vprintf(bfy_buffer* buf, char const* fmt, va_list args);
size_t bfy_buffer_add_hton_u8 (bfy_buffer* buf, uint8_t  addme);
size_t bfy_buffer_add_hton_u16(bfy_buffer* buf, uint16_t addme);
size_t bfy_buffer_add_hton_u32(bfy_buffer* buf, uint32_t addme);
size_t bfy_buffer_add_hton_u64(bfy_buffer* buf, uint64_t addme);

Of these functions, bfy_buffer_add() is the key: it copies the specified bytes into the free space at the end of the last page, marking that space as content. If not enough space is available, more is allocated. The number of bytes added is returned.

The rest are convenience wrappers: add_ch() adds a single character; add_printf() / add_vprintf() are printf-like functions that add to the end of the buffer, and add_hton_u*() will convert numbers into big-endian network byte order before ading them to he buffer.. Like bfy_buffer_add(), these all try to append to the end of the current page.

size_t bfy_buffer_add_buffer(bfy_buffer* buf, bfy_buffer* src);
size_t bfy_buffer_add_readonly(bfy_buffer* buf, const void* data, size_t len);
size_t bfy_buffer_add_reference(bfy_buffer* buf, const void* data, size_t len,
                                bfy_unref_cb* cb, void* user_data);

These functions add pre-existing content into the buffer by embedding it rather than duplicating it.

add_buffer() transfers the contents of one buffer to another.

add_reference() adds a new page that embeds content managed outside of bfy. When bfy is done with the page, the unref callback passed to add_reference() is called.

add_readonly() adds a new page that embeds read-only content.

Removing Data

size_t bfy_buffer_remove_range(bfy_buffer* buf, size_t begin, size_t end, void* setme);
size_t bfy_buffer_remove(bfy_buffer* buf, size_t len, void* setme);

char* bfy_buffer_remove_string(bfy_buffer* buf, size_t* len);
int bfy_buffer_remove_ntoh_u8 (bfy_buffer* buf, uint8_t* setme);
int bfy_buffer_remove_ntoh_u16(bfy_buffer* buf, uint16_t* setme);
int bfy_buffer_remove_ntoh_u32(bfy_buffer* buf, uint32_t* setme);
int bfy_buffer_remove_ntoh_u64(bfy_buffer* buf, uint64_t* setme);

bfy_buffer_remove_range() moves the [begin..end) subset of all the buffer's content from the buffer into setme and returns the number of bytes moved. bfy_buffer_remove() is a convenience helper to remove the buffer's first len bytes. Just as content is added with bfy_buffer_add*(), it is consumed with bfy_buffer_remove*().

The others are also helpers: bfy_buffer_remove_ntoh*() read numbers from big-endian network byte order into the host machine's byte order, and bfy_buffer_remove_string() will remove the buffer into a newly-allocated string.

int bfy_buffer_remove_buffer(bfy_buffer* buf, bfy_buffer* tgt, size_t len);

This moves the first len bytes of content from the source buffer to the target buffer. Unnecessary memory copying is avoided as much as possible.

size_t bfy_buffer_drain_range(bfy_buffer* buf, size_t begin, size_t end);
size_t bfy_buffer_drain(bfy_buffer* buf, size_t len);
size_t bfy_buffer_drain_all(bfy_buffer* buf);

bfy_buffer_drain_range() removes the [begin..end) subset of the buffer's content without copying it first. bfy_buffer_drain_all() and bfy_buffer_drain() are convenience helpers that drain the entire buffer or its first len bytes, respectively.

size_t bfy_buffer_copyout_range(bfy_buffer const* buf, size_t begin, size_t end, void* setme);
size_t bfy_buffer_copyout(bfy_buffer const* buf, size_t len, void* setme);

bfy_buffer_copyout_range() copies the [begin..end) subset of the buffer's contents into the provided setme buffer. bfy_buffer_copyout() is a convenience helper that copies the buffer's first len bytes.

Searching

int bfy_buffer_search_range(bfy_buffer const* buf,
                            size_t begin, size_t end,
                            void const* needle, size_t needle_len,
                            size_t* match);

int bfy_buffer_search_all(bfy_buffer const* buf,
                          void const* needle, size_t needle_len,
                          size_t* match);

int bfy_buffer_search(bfy_buffer const* buf, size_t len,
                          void const* needle, size_t needle_len,
                          size_t* match);

bfy_buffer_search_range() searches for a string inside the [begin..end) subset of the buffer's content. bfy_buffer_search_all() and bfy_buffer_search() are convenience helpers that search the entire buffer or its first len bytes, respectively.

Efficient Memory Management

Preallocating Space

If you know how much space you're going to use, it makes sense to tell bfy with bfy_buffer_ensure_space() so that it can be preallocated once in a single page before you start filling it with bfy_buffer_add*().

bfy_buffer_get_space_len() returns how much space is available.

size_t bfy_buffer_get_space_len(bfy_buffer const* buf);
int bfy_buffer_ensure_space(bfy_buffer* buf, size_t len);

Peek / Reserve / Commit

As an alternative to bfy_buffer_ensure_space() + bfy_buffer_add*(), the reserve/commit API lets you work directly a buffer's free space. bfy_buffer_reserve_space(len) preallocates len bytes of free space and returns its address. You can write directly to this memory or pass it along to a third party data generator, then commit your work to the buffer with bfy_buffer_commit_space(). It's OK to commit fewer bytes than you reserved, or even to skip committing altogether if you find that you didn't need the space after all.

bfy_buffer_peek_space() similar to bfy_buffer_reserve_space() but only returns the free space that's already allocated. This is not often needed but can be useful if you want to write as much content as possible on the current page before adding a new page. As with bfy_buffer_reserve_space(), you can commit some or all of your work when done.

It is an error to do anything that changes the buffer while you're still working on uncommitted memory. Changing the buffer can invalidate the pointers returned by peek/reserve.

struct bfy_iovec bfy_buffer_reserve_space(bfy_buffer* buf, size_t len);
struct bfy_iovec bfy_buffer_peek_space(bfy_buffer* buf);
size_t bfy_buffer_commit_space(bfy_buffer* buf, size_t len);

Contiguous / Non-contiguous Memory

As mentioned above in Concepts, buffers are made from a series of pages and a page is a contiguous block of memory.

If you need to view multiple pages' worth of memory as a contiguous block, bfy_buffer_make_contiguous(len) will ensure the first len bytes of content are in a single page. bfy_buffer_make_all_contiguous() is a convenience helper to make the entire buffer a single page.

Instead of the contiguous() functions, It is generally more efficient to walk through content with bfy_buffer_peek() or bfy_buffer_peek_all() instead, since that can be done without moving data into a single page.

User code won't often need it, but bfy_buffer_add_pagebreak() can be used to force a pagebreak between calls to bfy_buffer_add*().

void* bfy_buffer_make_contiguous(bfy_buffer* buf, size_t len);
void* bfy_buffer_make_all_contiguous(bfy_buffer* buf);
size_t bfy_buffer_peek(bfy_buffer const* buf, size_t size,
                       struct bfy_iovec* vec_out, size_t vec_len);
size_t bfy_buffer_peek_all(bfy_buffer const* buf,
                           struct bfy_iovec* vec_out, size_t vec_len);
int bfy_buffer_add_pagebreak(bfy_buffer* buf);

Comparison to evbuffer

libbuffy is inspired by libevent's evbuffer, so it's no surprise that they have similar APIs. But the software world is full of event loop tools, so if you're using glib or libuv or libev or libhv instead -- or none of the above! -- you may find libbuffy to be more accessible.

About

An embeddable, MIT-licensed, C-language, zero-dependency memory buffer class inspired by libevent's evbuffer. It consists of just a few files that can be dropped into your own project as-is.

Topics

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published