Skip to content

Commit

Permalink
Abstract the accurate payload storage from the datatype used.
Browse files Browse the repository at this point in the history
Up until now the accurate payload was always stored in a htable
(memory hash table) but that will not scale. We want to support multiple
storage classes for the accurate payload (e.g. the actual accurate data
accessed via the filename as key.) Things like Symas LMDB is a good
candidate as a key/value store. This patch implements the accurate data
using a class based abstraction that allows multiple implementations of
the storage. Currently it supports the old HTABLE and LMDB. The
prototype has some code that is not needed at this point as accurate
data is written only ones and after that never updated so we don't
strictly need an update method which is implemented for LMDB and is
unneeded for HTABLE as there you get internal memory you can manipulate.
The seen logic is moved into a simple bitmap that is part of the higher
level B_ACCURATE class so we don't need to update the actual accurate
payload when we only want to register that the item has been vistited.
Otherwise for things like LMDB we keep on updating the data in the LMDB
which is just to much overhead which we can easily work around with this
bitmap.
  • Loading branch information
Marco van Wieringen committed Apr 14, 2014
1 parent c20588a commit 54a57b0
Show file tree
Hide file tree
Showing 14 changed files with 1,164 additions and 234 deletions.
11 changes: 6 additions & 5 deletions src/filed/Makefile.in
Expand Up @@ -30,12 +30,13 @@ first_rule: all
dummy:

#
SVRSRCS = accurate.c authenticate.c backup.c compression.c \
crypto.c dir_cmd.c estimate.c fd_plugins.c filed_conf.c \
filed.c fileset.c heartbeat.c restore.c sd_cmds.c \
status.c verify_vol.c verify.c
SVRSRCS = accurate.c accurate_htable.c accurate_lmdb.c authenticate.c \
backup.c compression.c crypto.c dir_cmd.c estimate.c \
fd_plugins.c filed_conf.c filed.c fileset.c heartbeat.c \
restore.c sd_cmds.c status.c verify_vol.c verify.c
SVROBJS = $(SVRSRCS:.c=.o)

FDLIBS += @LMDB_LIBS@
FDLIBS += @CAP_LIBS@
FDLIBS += @ZLIB_LIBS_NONSHARED@
FDLIBS += @LZO_LIBS_NONSHARED@
Expand All @@ -44,7 +45,7 @@ FDLIBS += @AFS_LIBS_NONSHARED@
FDLIBS += @ACL_LIBS_NONSHARED@
FDLIBS += @XATTR_LIBS_NONSHARED@

INCLUDES += -I$(srcdir) -I$(basedir) -I$(basedir)/include
INCLUDES += -I$(srcdir) -I$(basedir) -I$(basedir)/include -I$(basedir)/lmdb

.SUFFIXES: .c .o
.PHONY:
Expand Down

0 comments on commit 54a57b0

Please sign in to comment.