Support for Intel SIMD Bitwise Operations #1533

Closed
wants to merge 605 commits into
from

Projects

None yet
@jonahharris
Contributor

This pull request IS NOT for direct addition, but for prompting discussion. Given the difference in performance, particularly on large bitstrings, is anyone interested in this?

erikdubbelboer and others added some commits Mar 30, 2012
@erikdubbelboer @antirez erikdubbelboer Added consts keyword where possible e04be06
@erikdubbelboer @antirez erikdubbelboer Fixed some spelling errors in the comments 04779bd
@antirez Scripting: redis.NIL to return nil bulk replies.
Lua arrays can't contain nil elements (see
http://www.lua.org/pil/19.1.html for more information), so Lua scripts
were not able to return a multi-bulk reply containing nil bulk
elements inside.

This commit introduces a special conversion: a table with just
a "nilbulk" field set to a boolean value is converted by Redis as a nil
bulk reply, but at the same time for Lua this type is not a "nil" so can
be used inside Lua arrays.

This type is also assigned to redis.NIL, so the following two forms
are equivalent and will be able to return a nil bulk reply as second
element of a three elements array:

    EVAL "return {1,redis.NIL,3}" 0
    EVAL "return {1,{nilbulk=true},3}" 0

The result in redis-cli will be:

    1) (integer) 1
    2) (nil)
    3) (integer) 3
e061d79
@antirez Scripting: add helper functions redis.error_reply() and redis.status_…
…reply().

A previous commit introduced Redis.NIL. This commit adds similar helper
functions to return tables with a single field set to the specified
string so that instead of using 'return {err="My Error"}' it is possible
to use a more idiomatic form:

    return redis.error_reply("My Error")
    return redis.status_reply("OK")
f1466e1
@agladysh

Please do not reference PiL I — it is about Lua 5.0, not 5.1. It is not a very good idea to encourage people to read it (unless they are to work with 5.0, which is not the case here), even if a given chapter is left intact in PiL II — it often leads to much confusion later on. A reference to a corresponding chapter in PiL II would be much better.

Owner

Hi @agladysh thanks for the hint, I was not aware at all about PiL I vs II, I'll take care the next time.

Please note the message at the top of each page of online PiL I.

Owner

So PiL II is not available in the web site as normal documentation? I guess I need to find a different resource then ;)

No, it is a commercial book (as PiL I was previously). I highly recommend you to buy and read it — I'd say it is one of best CS books ever. (But note that it is not very different from PiL I from that point of view.)

As for the references of the kind you use here, just quote the manual — it does contain this information.

Owner

I don't question the validity of the book and I'll be happy to purchase it and read it, thanks for the advice. However I don't want to link to resources that are not freely available on the web in commit comments, as everybody should be able to access it. Next time I'll put a note that what I'm writing is applicable to Lua 5.0 and with 5.1 things may be different.

Next time I'll put a note that what I'm writing is applicable to Lua 5.0 and with 5.1 things may be different.

IMHO such notice will be confusing. (Redis does not have anything to do with 5.0, and 5.0 is ages old.) I'd use the manual instead.

But it is not that big deal anyway.

@agladysh

It should be faster to store a reference to redis.NIL somewhere (e.g. in upvalue, in function environment or in registry) at the initialization time and simply compare table references here. (I.e. redis.NIL is considered to be a special unique token object.)

This is how (IMO) it is usually done, but the way you did here is OK as well. Note that the semantics is slightly different though — although not for "documented" use case.

(But see below.)

@agladysh

Erm. So, nilbulk becomes a reserved name, and I can't serialize a table if it contains that field? Not very clean IMO.

Owner

In Lua -> Redis conversion the only arrays that are valid to serialize are those with integer keys, not general Lua tables, so we are using only special-cases that are not serializable (that is, tables with non-numerical keys). Therefore there is no reserved word.

  1. I would put a comment about this in sources.

  2. Are you sure this will not change in future?

and others added some commits Oct 1, 2012
@antirez Revert "Scripting: redis.NIL to return nil bulk replies."
This reverts commit e061d79.

Conflicts:

	src/scripting.c
151b606
@antirez "SORT by nosort" (skip sorting) respect sorted set ordering.
When SORT is called with the option BY set to a string constant not
inclduing the wildcard character "*", there is no way to sort the output
so any ordering is valid. This allows the SORT internals to optimize its
work and don't really sort the output at all.

However it was odd that this option was not able to retain the natural
order of a sorted set. This feature was requested by users multiple
times as sometimes to call SORT with GET against sorted sets as a way to
mass-fetch objects can be handy.

This commit introduces two things:

1) The ability of SORT to return sorted sets elements in their natural
ordering when `BY nosort` is specified, accordingly to `DESC / ASC` options.
2) The ability of SORT to optimize this case further if LIMIT is passed
as well, avoiding to really fetch the whole sorted set, but directly
obtaining the specified range.

Because in this case the sorting is always deterministic, no
post-sorting activity is performed when SORT is called from a Lua
script.

This commit fixes issue #98.
2ba9627
@antirez "Timeout receiving bulk data" error message modified.
The new message now contains an hint about modifying the repl-timeout
configuration directive if the problem persists.

This should normally not be needed, because while the master generates
the RDB file it makes sure to send newlines to the replication channel
to prevent timeouts. However there are times when masters running on
very slow systems can completely stop for seconds during the RDB saving
process. In such a case enlarging the timeout value can fix the problem.

See issue #695 for an example of this problem in an EC2 deployment.
0c19880
@jokea @antirez jokea Force expire all timer events when system clock skew is detected.
When system time changes back, the timer will not worker properly
hence some core functionality of redis will stop working(e.g. replication,
bgsave, etc). See issue #633 for details.

The patch saves the previous time and when a system clock skew is detected,
it will force expire all timers.

Modiifed by @antirez: the previous time was moved into the eventLoop
structure to make sure the library is still thread safe as long as you
use different event loops into different threads (otherwise you need
some synchronization). More comments added about the reasoning at the
base of the patch, that's worth reporting here:

/* If the system clock is moved to the future, and then set back to the
 * right value, time events may be delayed in a random way. Often this
 * means that scheduled operations will not be performed soon enough.
 *
 * Here we try to detect system clock skews, and force all the time
 * events to be processed ASAP when this happens: the idea is that
 * processing events earlier is less dangerous than delaying them
 * indefinitely, and practice suggests it is. */
e480c27
@antirez Include time.h in ae.c as we now use time(). e2f2dab
@antirez Warn when configured maxmemory value seems odd.
This commit warns the user with a log at "warning" level if:

1) After the server startup the maxmemory limit was found to be < 1MB.
2) After a CONFIG SET command modifying the maxmemory setting the limit
is set to a value that is smaller than the currently used memory.

The behaviour of the Redis server is unmodified, and this wil not make
the CONFIG SET command or a wrong configuration in redis.conf less
likely to create problems, but at least this will make aware most users
about a possbile error they committed without resorting to external
help.

However no warning is issued if, as a result of loading the AOF or RDB
file, we are very near the maxmemory setting, or key eviction will be
needed in order to go under the specified maxmemory setting. The reason
is that in servers configured as a cache with an aggressive
maxmemory-policy most of the times restarting the server will cause this
condition to happen if persistence is not switched off.

This fixes issue #429.
05e06e1
@antirez Hash function switched to murmurhash2.
The previously used hash function, djbhash, is not secure against
collision attacks even when the seed is randomized as there are simple
ways to find seed-independent collisions.

The new hash function appears to be safe (or much harder to exploit at
least) in this case, and has better distribution.

Better distribution does not always means that's better. For instance in
a fast benchmark with "DEBUG POPULATE 1000000" I obtained the following
results:

    1.6 seconds with djbhash
    2.0 seconds with murmurhash2

This is due to the fact that djbhash will hash objects that follow the
pattern `prefix:<id>` and where the id is numerically near, to near
buckets. This improves the locality.

However in other access patterns with keys that have no relation
murmurhash2 has some (apparently minimal) speed advantage.

On the other hand a better distribution should significantly
improve the quality of the distribution of elements returned with
dictGetRandomKey() that is used in SPOP, SRANDMEMBER, RANDOMKEY, and
other commands.

Everything considered, and under the suspect that this commit fixes a
security issue in Redis, we are switching to the new hash function.
If some serious speed regression will be found in the future we'll be able
to step back easiliy.

This commit fixes issue #663.
99c3338
@antirez Merge pull request #544 from dvirsky/2.6
fixed install script to rewrite the default config
a6305f1
@antirez Redis 2.6.0 RC8 (2.5.14) 500dddc
@antirez CONTRIBUTING file updated. f4eb4b3
@antirez Allow AUTH when Redis is busy because of timedout Lua script.
If the server is password protected we need to accept AUTH when there is
a server busy (-BUSY) condition, otherwise it will be impossible to send
SHUTDOWN NOSAVE or SCRIPT KILL.

This fixes issue #708.
0e25c0c
@antirez Merge remote-tracking branch 'origin/2.6' into 2.6 c3ff470
@antirez Fix MULTI / EXEC rendering in MONITOR output.
Before of this commit it used to be like this:

MULTI
EXEC
... actual commands of the transaction ...

Because after all that is the natural order of things. Transaction
commands are queued and executed *only after* EXEC is called.

However this makes debugging with MONITOR a mess, so the code was
modified to provide a coherent output.

What happens is that MULTI is rendered in the MONITOR output as far as
possible, instead EXEC is propagated only after the transaction is
executed, or even in the case it fails because of WATCH, so in this case
you'll simply see:

MULTI
EXEC

An empty transaction.
2164523
@moreaki

nit: s/MUTLI/MULTI/

otherwise looks like a sane fix.

NanXiao and others added some commits Oct 10, 2012
@NanXiao @antirez NanXiao Update src/redis-benchmark.c
The code of current implementation:

if (c->pending == 0) clientDone(c);
In clientDone function, the c's memory has been freed, then the loop will continue: while(c->pending). The memory of c has been freed now, so c->pending is invalid (c is an invalid pointer now), and this will cause memory dump in some platforams(eg: Solaris).

So I think the code should be modified as:
if (c->pending == 0)
{
clientDone(c);
break;
}
and this will not lead to while(c->pending).
a03c327
@antirez Differentiate SCRIPT KILL error replies.
When calling SCRIPT KILL currently you can get two errors:

* No script in timeout (busy) state.
* The script already performed a write.

It is useful to be able to distinguish the two errors, but right now both
start with "ERR" prefix, so string matching (that is fragile) must be used.

This commit introduces two different prefixes.

-NOTBUSY and -UNKILLABLE respectively to reply with an error when no
script is busy at the moment, and when the script already executed a
write operation and can not be killed.
ab55180
@antirez Default memory limit for 32bit instanced moved from 3.5 GB to 3 GB.
In some system, notably osx, the 3.5 GB limit was too far and not able
to prevent a crash for out of memory. The 3 GB limit works better and it
is still a lot of memory within a 4 GB theorical limit so it's not going
to bore anyone :-)

This fixes issue #711
a25b25f
@ripcurld0 @antirez ripcurld0 redis-check-dump now understands dumps produced by Redis 2.6
(Commit message from @antirez as it was missign in the original commits,
also the patch was modified a bit to still work with 2.4 dumps and to
avoid if expressions that are always true due to checked types range)

This commit changes redis-check-dump to account for new encodings and
for the new MSTIME expire format. It also refactors the test for valid
type into a function.

The code is still compatible with Redis 2.4 generated dumps.

This fixes issue #709.
1614458
@antirez Greg Hurrell Fix (cosmetic) typos in dict.h a61705d
@antirez A filed called slave_read_only added in INFO output.
This was an important information missing from the INFO output in the
replication section.

It obviously reflects if the slave is read only or not.
99d7dbe
@antirez Redis 2.6.0 5eec376
@antirez Update memory peak stats while loading RDB / AOF. 1eb9145
@antirez Fix compilation on Linux kernels or glibc versions lacking sync_file_…
…range().

This fixes issue #667.

Many thanks to Didier Spezia for the fix.
be407c0
@antirez Redis 2.6.1 2b5fc52
@openbaas @antirez openbaas patch config.h for sync_file_range b25b0dc
@antirez Marginally more robust glibc version test for sync_file_range detection. 9b3b110
@antirez Ctrl+w support in linenoise. 3b1a41b
@antirez Redis 2.6.2 0dd6241
@antirez help.h update (adds bitop, bitcount, evalsha...) ec0f483
@antirez No longer used macro rdbIsOpcode() removed. 5bf0997
@antirez Invert two sides of if expression in SET to avoid a lookup.
Because of the short circuit behavior of && inverting the two sides of
the if expression avoids an hash table lookup if the non-EX variant of
SET is called.

Thanks to Weibin Yao (@yaoweibin on github) for spotting this.
4d9bd53
@yamt @antirez yamt fix a typo in a comment 47ff443
@cofyc cofyc fix typo in comments (redis.c, networking.c) ed44a74
@antirez 32 bit build fixed on Linux.
It failed because of the way jemalloc was compiled (without passing the
right flags to make, but just to configure). Now the same set of flags
are also passed to the make command, fixing the issue.

This fixes issue #744
4917a6a
@antirez Unix socket clients properly displayed in MONITOR and CLIENT LIST.
This also fixes issue #745.
b6ffa85
@antirez Merge pull request #743 from Cofyc/fixtypo
fix typo in comments (redis.c, networking.c)
cc017c5
@antirez More robust handling of AOF rewrite child.
After the wait3() syscall we used to do something like that:

    if (pid == server.rdb_child_pid) {
        backgroundSaveDoneHandler(exitcode,bysignal);
    } else {
        ....
    }

So the AOF rewrite was handled in the else branch without actually
checking if the pid really matches. This commit makes the check explicit
and logs at WARNING level if the pid returned by wait3() does not match
neither the RDB or AOF rewrite child.
36f026a
@run @antirez run fix a typo in redis.h line 595 comment 3f00667
@antirez Redis 2.6.3 89514f8
@antirez BSD license added to every C source and header file. d288ee6
@antirez Make clear that contributing code to the Redis project means to relea…
…se it under the terms of the BSD license.
8e30092
@antirez Redis 2.6.4 2861cd8
@antirez Copyright date fixed in COPYING file. 5a8234d
@antirez MIGRATE: fix fd leak due to missing close on error. ae3aeca
@antirez MIGRATE timeout should be in milliseconds.
While it is documented that the MIGRATE timeout is in milliseconds, it
was in seconds instead. This commit fixes the problem.
26a4850
@antirez MIGRATE: fix default timeout to 1000 milliseconds.
When a timeout <= 0 is provided we set a default timeout of 1 second.
It was set to 1 millisecond for an error resulting from a recent change.
d85a09d
@antirez zmalloc: kill unused __size parameter in update_zmalloc_stat_alloc() …
…macro.
681c191
@antirez zmalloc_get_private_dirty() function added (Linux only).
For non Linux systmes it just returns 0.

This function is useful to estimate copy-on-write because of childs
saving stuff on disk.
e95ca66
@antirez Children creating AOF or RDB files now report memory used by COW.
Finally Redis is able to report the amount of memory used by
copy-on-write while saving an RDB or writing an AOF file in background.

Note that this information is currently only logged (at NOTICE level)
and not shown in INFO because this is less trivial (but surely doable
with some minor form of interprocess communication).

The reason we can't capture this information on the parent before we
call wait3() is that the Linux kernel will release the child memory
ASAP, and only retain the minimal state for the process that is useful
to report the child termination to the parent.

The COW size is obtained by summing all the Private_Dirty fields found
in the "smap" file inside the proc filesystem for the process.

All this is Linux specific and is not available on other systems.
5ab4151
@antirez Safer handling of MULTI/EXEC on errors.
After the transcation starts with a MULIT, the previous behavior was to
return an error on problems such as maxmemory limit reached. But still
to execute the transaction with the subset of queued commands on EXEC.

While it is true that the client was able to check for errors
distinguish QUEUED by an error reply, MULTI/EXEC in most client
implementations uses pipelining for speed, so all the commands and EXEC
are sent without caring about replies.

With this change:

1) EXEC fails if at least one command was not queued because of an
error. The EXECABORT error is used.
2) A generic error is always reported on EXEC.
3) The client DISCARDs the MULTI state after a failed EXEC, otherwise
pipelining multiple transactions would be basically impossible:
After a failed EXEC the next transaction would be simply queued as
the tail of the previous transaction.
41f0f92
@antirez Test: MULTI/EXEC tests moved into multi.tcl. 37d978d
@antirez Test: make sure EXEC fails after previous transaction errors. 52bd3d8
@antirez Test: MULTI state is cleared after EXECABORT error. 2b45ef0
@antirez Fix integer overflow in zunionInterGenericCommand().
This fixes issue #761.
de00a5a
@antirez EVALSHA is now case insensitive.
EVALSHA used to crash if the SHA1 was not lowercase (Issue #783).
Fixed using a case insensitive dictionary type for the sha -> script
map used for replication of scripts.
9120275
@antirez Redis 2.6.5 3ccfb5a
@antirez Jemalloc updated to version 3.2.0. 21b2691
@antirez Redis 2.6.6 b41f4e5
@antirez Make an EXEC test more latency proof. ed70955
@antirez Client should not block multiple times on the same key.
Sending a command like:

BLPOP foo foo foo foo 0

Resulted into a crash before this commit since the client ended being
inserted in the waiting list for this key multiple times.
This resulted into the function handleClientsBlockedOnLists() to fail
because we have code like that:

    if (de) {
        list *clients = dictGetVal(de);
        int numclients = listLength(clients);

        while(numclients--) {
            listNode *clientnode = listFirst(clients);

            /* server clients here... */
        }
    }

The code to serve clients used to remove the served client from the
waiting list, so if a client is blocking multiple times, eventually the
call to listFirst() will return NULL or worse will access random memory
since the list may no longer exist as it is removed by the function
unblockClientWaitingData() if there are no more clients waiting for this
list.

To avoid making the rest of the implementation more complex, this commit
modifies blockForKeys() so that a client will be put just a single time
into the waiting list for a given key.

Since it is Saturday, I hope this fixes issue #801.
cac49a9
@antirez Test: regression for issue #801. 6bdcd50
@antirez Blocking POP: use a dictionary to store keys clinet side.
To store the keys we block for during a blocking pop operation, in the
case the client is blocked for more data to arrive, we used a simple
linear array of redis objects, in the blockingState structure:

    robj **keys;
    int count;

However in order to fix issue #801 we also use a dictionary in order to
avoid to end in the blocked clients queue for the same key multiple
times with the same client.

The dictionary was only temporary, just to avoid duplicates, but since
we create / destroy it there is no point in doing this duplicated work,
so this commit simply use a dictionary as the main structure to store
the keys we are blocked for. So instead of the previous fields we now
just have:

    dict *keys;

This simplifies the code and reduces the work done by the server during
a blocking POP operation.
54b08c8
@antirez Test: fixed osx "leaks" support in test.
Due to changes in recent releases of osx leaks utility, the osx leak
detection no longer worked. Now it is fixed in a way that should be
backward compatible.
a2b3fff
@antirez Memory leak fixed: release client's bpop->keys dictionary.
Refactoring performed after issue #801 resolution (see commit
2f87cf8) introduced a memory leak that
is fixed by this commit.

I simply forgot to free the new allocated dictionary in the client
structure trusting the output of "make test" on OSX.

However due to changes in the "leaks" utility the test was no longer
testing memory leaks. This problem was also fixed.

Fortunately the CI test running at ci.redis.io spotted the bug in the
valgrind run.

The leak never ended into a stable release.
984f6ed
@bmcmanus @antirez bmcmanus Issue 804 Add Default-Start and Default-Stop LSB tags for RedHat star…
…tup and update-rc.d compatability.
6c9897f
@antirez Redis 2.6.7 d9301f0
@antirez Fix config.h endianess detection to work on Linux / PPC64.
Config.h performs endianess detection including OS-specific headers to
define the endianess macros, or when this is not possible, checking the
processor type via ifdefs.

Sometimes when the OS-specific macro is included, only __BYTE_ORDER is
defined, while BYTE_ORDER remains undefined. There is code at the end of
config.h endianess detection in order to define the macros without the
underscore, but it was not working correctly.

This commit fixes endianess detection fixing Redis on Linux / PPC64 and
possibly other systems.
d64a9cf
@antirez Patrick TJ McPhee Define _XOPEN_SOURCE appropriately on NetBSD. 8e0a9f6
@antirez Added missing license and copyright in deps/hiredis. 886c9ec
@antirez CONTRIBUTING updated with request to add BSD license. d8a0fdd
@antirez Fix overflow in mstime() in redis-cli and benchmark.
The problem does not exist in the Redis server implementation of mstime()
but is only limited to redis-cli and redis-benchmark.

Thix fixes issue #839.
4468ba2
@antirez ae.c: set errno when error is not a failing syscall.
In this way the caller is able to perform better error checking or to
use strerror() without the risk of meaningless error messages being
displayed.
1f8ad7a
@antirez Better error reporting when fd event creation fails. c1c3e9b
@antirez Multiple fixes for EVAL (issue #872).
1) The event handler was no restored after a timeout condition if the
   command was eventually executed with success.
2) The command was not converted to EVAL in case of errors in the middle
   of the execution.
3) Terrible duplication of code without any apparent reason.
46dc3fb
@antirez Test: added regression for issue #872. b70b9c6
@antirez Comment in the call() function clarified a bit. 023cbc3
@antirez Redis 2.6.8 14cc004
@antirez Makefile.dep updated. 78b175a
@antirez Undo slave-master handshake when SLAVEOF sets a new slave.
Issue #828 shows how Redis was not correctly undoing a non-blocking
connection attempt with the previous master when the master was set to a
new address using the SLAVEOF command.

This was also a result of lack of refactoring, so now there is a
function to cancel the non blocking handshake with the master.
The new function is now used when SLAVEOF NO ONE is called or when
SLAVEOF is used to set the master to a different address.
27abaa2
@antirez CLIENT GETNAME and CLIENT SETNAME introduced.
Sometimes it is much simpler to debug complex Redis installations if it
is possible to assign clients a name that is displayed in the CLIENT
LIST output.

This is the case, for example, for "leaked" connections. The ability to
provide a name to the client makes it quite trivial to understand what
is the part of the code implementing the client not releasing the
resources appropriately.

Behavior:

    CLIENT SETNAME: set a name for the client, or remove the current
                    name if an empty name is set.
    CLIENT GETNAME: get the current name, or a nil.
    CLIENT LIST: now displays the client name if any.

Thanks to Mark Gravell for pushing this idea forward.
c5f23ca
@antirez Typo fixed, ASCI -> ASCII. 1b2ac3b
@antirez Tests for CLIENT GETNAME/SETNAME. f9c1263
@antirez redis-cli: save an RDB dump from remote server to local file. 52f6a1d
@antirez Redis 2.6.9 21159d5
@nparry @antirez nparry redis-cli --rdb fails if server sends a ping
Redis pings slaves in "pre-synchronization stage" with newlines. (See
https://github.com/antirez/redis/blob/2.6.9/src/replication.c#L814)
However, redis-cli does not expect this - it sees the newline as the end
of the bulk length line, and ends up returning 0 as bulk the length.
This manifests as the following when running redis-cli:

    $ ./src/redis-cli --rdb some_file
    SYNC sent to master, writing 0 bytes to 'some_file'
    Transfer finished with success.

With this commit, we just ignore leading newlines while reading the bulk
length line.

To reproduce the problem, load enough data into Redis so that the
preparation of the RDB snapshot takes long enough for a ping to occur
while redis-cli is waiting for the data.
0f4dbd9
@bitterb @antirez bitterb Fix an error reply for CLIENT command a732d86
@badboy @antirez badboy Always exit if connection fails.
This avoids unnecessary core dumps. Fixes antirez/redis#894
5e2aa2c
@openbaas @antirez openbaas redis-cli prompt bug fix c2b43ea
@guiquanz @antirez guiquanz Fixed many typos. 560e049
@antirez Slowlog: don't log EXEC but just the executed commands.
The Redis Slow Log always used to log the slow commands executed inside
a MULTI/EXEC block. However also EXEC was logged at the end, which is
perfectly useless.

Now EXEC is no longer logged and a test was added to test this behavior.

This fixes issue #759.
dbde1d8
@antirez Clear server.shutdown_asap on failed shutdown.
When a SIGTERM is received Redis schedules a shutdown. However if it
fails to perform the shutdown it must be clear the shutdown_asap flag
otehrwise it will try again and again possibly making the server
unusable.
9e5b701
@antirez Whitelist SIGUSR1 to avoid auto-triggering errors.
This commit fixes issue #875 that was caused by the following events:

1) There is an active child doing BGSAVE.
2) flushall is called (or any other condition that makes Redis killing
the saving child process).
3) An error is sensed by Redis as the child exited with an error (killed
by a singal), that stops accepting write commands until a BGSAVE happens
to be executed with success.

Whitelisting SIGUSR1 and making sure Redis always uses this signal in
order to kill its own children fixes the issue.
da54022
@antirez Additionally two typos fixed thanks to @jodal a154908
and others added some commits Mar 12, 2013
@antirez Set default for stop_writes_on_bgsave_err in initServerConfig().
It was placed for error in initServer() that's called after the
configuation is already loaded, causing issue #1000.
18d16f8
@djanowski @antirez djanowski Abort when opening the RDB file results in an error other than ENOENT.
This fixes cases where the RDB file does exist but can't be accessed for
any reason. For instance, when the Redis process doesn't have enough
permissions on the file.
b9f8c2a
@antirez rdbLoad(): rework code to save vertical space. 189e865
@antirez Test: check that Redis starts empty without an RDB file. 684b615
@antirez Test: more RDB loading checks.
A test for issue #1001 is included.
ba3c549
@antirez Test: make sure broken RDB checksum is detected. c642559
@antirez Replication: master_link_down_since_seconds initial value should be h…
…uge.

server.repl_down_since used to be initialized to the current time at
startup. This is wrong since the replication never started. Clients
testing this filed to check if data is uptodate should never believe
data is recent if we never ever connected to our master.
4f8b18f
@antirez redis-cli --stat, stolen from redis-tools.
Redis-tools is a connection of tools no longer mantained that was
intented as a way to economically make sense of Redis in the pre-vmware
sponsorship era. However there was a nice redis-stat utility, this
commit imports one of the functionalities of this tool here in redis-cli
as it seems to be pretty useful.

Usage: redis-cli --stat

The output is similar to vmstat in the format, but with Redis specific
stuff of course.

From the point of view of the monitored instance, only INFO is used in
order to grab data.
5576a28
@antirez Test: obuf-limits test false positive removed.
Fixes #621.
dee8d84
@NanXiao @antirez NanXiao Update config.c
Fix bug in configGetCommand function: get correct masterauth value.
ef1cf15
@antirez Allow SELECT while loading the DB.
Fixes issue #1024.
6818905
@antirez Transactions: use the propagate() API to propagate MULTI.
The behavior is the same, but the code is now cleaner and uses the
proper interface instead of dealing directly with AOF/replication
functions.
d1369c3
@antirez Transactions: use discardTransaction() in EXEC implementation. 889b017
@antirez Transactions: propagate MULTI/EXEC only when needed.
MULTI/EXEC is now propagated to the AOF / Slaves only once we encounter
the first command that is not a read-only one inside the transaction.

The old behavior was to always propagate an empty MULTI/EXEC block when
the transaction was composed just of read only commands, or even
completely empty. This created two problems:

1) It's a bandwidth waste in the replication link and a space waste
   inside the AOF file.

2) We used to always increment server.dirty to force the propagation of
   the EXEC command, resulting into triggering RDB saves more often
   than needed.

Note: even read-only commands may also trigger writes that will be
propagated, when we access a key that is found expired and Redis will
synthesize a DEL operation. However there is no need for this to stay
inside the transaction itself, but only to be ordered.

So for instance something like:

    MULTI
    GET foo
    SET key zap
    EXEC

May be propagated into:

    DEL foo
    MULTI
    SET key zap
    EXEC

While the DEL is outside the transaction, the commands are delivered in
the right order and it is not possible for other commands to be inserted
between DEL and MULTI.
611dcb5
@antirez Flag PUBLISH as read-only in the command table. 6cb7860
@antirez Test: new functions to capture and analyze the replication stream. fc78d97
@antirez Test: Restore DB back to 9 after testing MULTI/EXEC with DB 5. 26319d0
@antirez Test: test replication of MULTI/EXEC. f8791a1
@antirez Test: read_from_replication_stream, workaround for 2.6 1862239
@antirez DEBUG set-active-expire added.
We need the ability to disable the activeExpireCycle() (active
expired key collection) call for testing purposes.
02b9a72
@antirez Test: verify that lazy-expire works. 7ff729e
@antirez Better DEBUG error message when num of arguments is wrong. 47911b8
@antirez Test: regression test for issue #1026. dc0f137
@antirez EXPIRE should not resurrect keys. Issue #1026. e8a1a16
@antirez Extended SET command implemented (issue #931). d785413
@antirez Test: Extended SET tests. 4243483
@openbaas @antirez openbaas Support for case unsensitive SET options. 062f60a
@antirez Redis 2.6.12 7da5980
@antirez Throttle BGSAVE attempt on saving error.
When a BGSAVE fails, Redis used to flood itself trying to BGSAVE at
every next cron call, that is either 10 or 100 times per second
depending on configuration and server version.

This commit does not allow a new automatic BGSAVE attempt to be
performed before a few seconds delay (currently 5).

This avoids both the auto-flood problem and filling the disk with
logs at a serious rate.

The five seconds limit, considering a log entry of 200 bytes, will use
less than 4 MB of disk space per day that is reasonable, the sysadmin
should notice before of catastrofic events especially since by default
Redis will stop serving write queries after the first failed BGSAVE.

This fixes issue #849
ed2d988
@xiaost7
Contributor

describes like

[BUGFIX] Shutdown at startup if RDB is there but can't be opened.

is better :)

and others added some commits Apr 3, 2013
@antirez Make rio.c comment 80-columns friendly. 63978d3
@antirez Test: remove useless statements and comments from test default config. 9f2e6ac
@antirez redis-cli: --latency-history mode implemented. 92a7b01
@antirez redis-cli: raise error on bad command line switch.
Previously redis-cli never tried to raise an error when an unrecognized
switch was encountered, as everything after the initial options is to be
transmitted to the server.

However this is too liberal, as there are no commands starting with "-".
So the new behavior is to produce an error if there is an unrecognized
switch starting with "-". This should not break past redis-cli usages
but should prevent broken options to be silently discarded.

As far the first token not starting with "-" is encountered, all the
rest is considered to be part of the command, so you cna still use
strings starting with "-" as values, like in:

    redis-cli --port 6380 set foo --my-value
4086151
@antirez Redis/Jemalloc Gitignore were too aggressive.
Redis gitignore was too aggressive since simply broken.

Jemalloc gitignore was too agressive because it is conceived to just
keep the files that allow to generate all the rest in development
environments (so for instance the "configure" file is excluded).
5bf9b53
@antirez More explicit panic message on out of memory. 172eac5
@antirez Test: split conceptually unrelated comments in RDB test. 4848af1
@antirez Sentinel: turn old master into a slave when it comes back. bac29c8
@antirez Test: fix RDB test checking file permissions.
When the test is executed using the root account, setting the permission
to 222 does not work as expected, as root can read files with 222
permission.

Now we skip the test if root is detected.

This fixes issue #1034 and the duplicated #1040 issue.

Thanks to Jan-Erik Rediger (@badboy on Github) for finding a way to reproduce the issue.
cf7b2c3
@antirez rio.c: added ability to fdatasync() from time to time while writing. 8b41e19
@antirez AOF: sync data on disk every 32MB when rewriting.
This prevents the kernel from putting too much stuff in the output
buffers, doing too heavy I/O all at once. So the goal of this commit is
to split the disk pressure due to the AOF rewrite process into smaller
spikes.

Please see issue #1019 for more information.
c735116
@antirez Config option to turn AOF rewrite incremental fsync on/off. 967ae8c
@antirez Sentinel: always redirect on master->slave transition.
Sentinel redirected to the master if the instance changed runid or it
was the first time we got INFO, and a role change was detected from
master to slave.

While this is a good idea in case of slave->master, since otherwise we
could detect a failover without good reasons just after a reboot with a
slave with a wrong configuration, in the case of master->slave
transition is much better to always perform the redirection for the
following reasons:

1) A Sentinel may go down for some time. When it is back online there is
no other way to understand there was a failover.
2) Pointing clients to a slave seems to be always the wrong thing to do.
3) There is no good rationale about handling things differently once an
instance is rebooted (runid change) in that case.
34a57a5
@antirez Lua updated to version 5.1.5. 395e6a5
@antirez Sentinel: only demote old master into slave under certain conditions.
We used to always turn a master into a slave if the DEMOTE flag was set,
as this was a resurrecting master instance.

However the following race condition is possible for a Sentinel that
got partitioned or internal issues (tilt mode), and was not able to
refresh the state in the meantime:

1) Sentinel X is running, master is instance "A".
3) "A" fails, sentinels will promote slave "B" as master.
2) Sentinel X goes down because of a network partition.
4) "A" returns available, Sentinels will demote it as a slave.
5) "B" fails, other Sentinels will promote slave "A" as master.
6) At this point Sentinel X comes back.

When "X" comes back he thinks that:

"B" is the master.
"A" is the slave to demote.

We want to avoid that Sentinel "X" will demote "A" into a slave.
We also want that Sentinel "X" will detect that the conditions changed
and will reconfigure itself to monitor the right master.

There are two main ways for the Sentinel to reconfigure itself after
this event:

1) If "B" is reachable and already configured as a slave by other
sentinels, "X" will perform a redirection to "A".
2) If there are not the conditions to demote "A", the fact that "A"
reports to be a master will trigger a failover detection in "X", that
will end into a reconfiguraiton to monitor "A".

However if the Sentinel was not reachable, its state may not be updated,
so in case it titled, or was partiitoned from the master instance of the
slave to demote, the new implementation waits some time (enough to
guarantee we can detect the new INFO, and new DOWN conditions).

If after some time still there are not the right condiitons to demote
the instance, the DEMOTE flag is cleared.
fb0d08e
@antirez Sentinel: more sensible delay in master demote after tilt. 59ff2fe
@antirez Sentinel: changes to tilt mode.
Tilt mode was too aggressive (not processing INFO output), this
resulted in a few problems:

1) Redirections were not followed when in tilt mode. This opened a
   window to misinform clients about the current master when a Sentinel
   was in tilt mode and a fail over happened during the time it was not
   able to update the state.

2) It was possible for a Sentinel exiting tilt mode to detect a false
   fail over start, if a slave rebooted with a wrong configuration
   about at the same time. This used to happen since in tilt mode we
   lose the information that the runid changed (reboot).

   Now instead the Sentinel in tilt mode will still remove the instance
   from the list of slaves if it changes state AND runid at the same
   time.

Both are edge conditions but the changes should overall improve the
reliability of Sentinel.
4f38d03
@antirez Redis 2.6.13 d964979
@openbaas @antirez openbaas Fix AOF bug: expire could be removed from key on AOF rewrite.
There was a race condition in the AOF rewrite code that, with bad enough
timing, could cause a volatile key just about to expire to be turned
into a non-volatile key. The bug was never reported to cause actualy
issues, but was found analytically by an user in the Redis mailing list:

https://groups.google.com/forum/?fromgroups=#!topic/redis-db/Kvh2FAGK4Uk

This commit fixes issue #1079.
b37512a
@soveran @antirez soveran Reset masterauth if an empty string is configured. 736220b
@antirez CONFIG SET server.masterauth aesthetic change.
This is just to make the code exactly like the above instance used for
requirepass. No actual change nor the original code violated the Redis
coding style.
6a947b1
@antirez Added more info about 32 bit builds into README. 71b4422
@july2993 @antirez july2993 in 32bit machine, popcount don't work with a input string length up t…
…o 512 MB,

bitcount commant may return negtive integer with string length more than 256 MB
6251c3e
@antirez Test: various issues with the replication-4.tcl test fixed.
The test actually worked, but vars for master and slave were inverted
and sometimes used incorrectly.
2c93d34
@antirez redis-cli: help.h updated. feffd56
@antirez Binary safe dump of object content in redisLogObjectDebugInfo(). a73da27
@antirez Allow writes from scripts called by AOF loading in read-only slaves.
This fixes issue #1163
8e63fd0
@antirez Test: regression test for #1163. ee92c04
@antirez Sentinel: parse new INFO replication output correctly.
Sentinel was not able to detect slaves when connected to a very recent
version of Redis master since a previos non-backward compatible change
to INFO broken the parsing of the slaves ip:port INFO output.

This fixes issue #1164
252f0b6
@antirez Redis 2.6.14 f2f2b4e
@antirez Fixed typo in 2.6.14 changelog. verison -> version. 16ddbb7
@quiver
quiver commented on f2f2b4e Jun 21, 2013
  • * Lua scripting + Replication + AOF in slaves problem (see Issue #1164).

Do you mean #1163, instead of #1164.

and others added some commits Aug 5, 2013
@antirez Draft #1 of a new expired keys collection algorithm.
The main idea here is that when we are no longer to expire keys at the
rate the are created, we can't block more in the normal expire cycle as
this would result in too big latency spikes.

For this reason the commit introduces a "fast" expire cycle that does
not run for more than 1 millisecond but is called in the beforeSleep()
hook of the event loop, so much more often, and with a frequency bound
to the frequency of executed commnads.

The fast expire cycle is only called when the standard expiration
algorithm runs out of time, that is, consumed more than
REDIS_EXPIRELOOKUPS_TIME_PERC of CPU in a given cycle without being able
to take the number of already expired keys that are yet not collected
to a number smaller than 25% of the number of keys.

You can test this commit with different loads, but a simple way is to
use the following:

Extreme load with pipelining:

redis-benchmark -r 100000000 -n 100000000  \
        -P 32 set ele:rand:000000000000 foo ex 2

Remove the -P32 in order to avoid the pipelining for a more real-world
load.

In another terminal tab you can monitor the Redis behavior with:

redis-cli -i 0.1 -r -1 info keyspace

and

redis-cli --latency-history

Note: this commit will make Redis printing a lot of debug messages, it
is not a good idea to use it in production.
ab78261
@antirez Darft #2 for key collection algo: more improvements.
This commit makes the fast collection cycle time configurable, at
the same time it does not allow to run a new fast collection cycle
for the same amount of time as the max duration of the fast
collection cycle.
d9a4fd0
@antirez Remove dead code and fix comments for new expire code. 97f5e86
@antirez Some activeExpireCycle() refactoring. 1bcf296
@antirez activeExpireCycle(): fix about fast cycle early start.
We don't want to repeat a fast cycle too soon, the previous code was
broken, we need to wait two times the period *since* the start of the
previous cycle in order to avoid there is an even space between cycles:

.-> start                   .-> second start
|                           |
+-------------+-------------+--------------+
| first cycle |    pause    | second cycle |
+-------------+-------------+--------------+

The second and first start must be PERIOD*2 useconds apart hence the *2
in the new code.
2e4ec79
@antirez Add per-db average TTL information in INFO output.
Example:

db0:keys=221913,expires=221913,avg_ttl=655

The algorithm uses a running average with only two samples (current and
previous). Keys found to be expired are considered at TTL zero even if
the actual TTL can be negative.

The TTL is reported in milliseconds.
487e665
@antirez redis-benchmark: ability to SELECT a specifid db number. d2d3ac3
@antirez redis-benchmark: fix db selection when :rand: feature is used. d5a4c3e
@antirez redis-benchmark: max pipeline length hardcoded limit removed. 79635cb
@antirez redis-benchmark: fix memory leak introduced by 346256f 4343f0b
@badboy @antirez badboy Little typo 702f3af
@antirez redis-benchmark: replace snprintf()+memcpy with faster code.
This change was profiler-driven, but the actual effect is hard to
measure in real-world redis benchmark runs.
f4f9313
@antirez redis-benchmark: changes to random arguments substitution.
Before this commit redis-benchmark supported random argumetns in the
form of :rand:000000000000. In every string of that form, the zeros were
replaced with a random number of 12 digits at every command invocation.

However this was far from perfect as did not allowed to generate simply
random numbers as arguments, there was always the :rand: prefix.

Now instead every argument in the form __rand_int__ is replaced with a
12 digits number. Note that "__rand_int__" is 12 characters itself.

In order to implement the new semantic, it was needed to change a few
thigns in the internals of redis-benchmark, as new clients are created
cloning old clients, so without a stable prefix such as ":rand:" the old
way of cloning the client was no longer able to understand, from the old
command line, what was the position of the random strings to substitute.

Now instead a client structure is passed as a reference for cloning, so
that we can directly clone the offsets inside the command line.
1c4e301
@antirez Fix sdsempty() prototype in sds.h. db1e135
@antirez dict.c iterator API misuse protection.
dict.c allows the user to create unsafe iterators, that are iterators
that will not touch the dictionary data structure in any way, preventing
copy on write, but at the same time are limited in their usage.

The limitation is that when itearting with an unsafe iterator, no call
to other dictionary functions must be done inside the iteration loop,
otherwise the dictionary may be incrementally rehashed resulting into
missing elements in the set of the elements returned by the iterator.

However after introducing this kind of iterators a number of bugs were
found due to misuses of the API, and we are still finding
bugs about this issue. The bugs are not trivial to track because the
effect is just missing elements during the iteartion.

This commit introduces auto-detection of the API misuse. The idea is
that an unsafe iterator has a contract: from initialization to the
release of the iterator the dictionary should not change.

So we take a fingerprint of the dictionary state, xoring a few important
dict properties when the unsafe iteartor is initialized. We later check
when the iterator is released if the fingerprint is still the same. If it
is not, we found a misuse of the iterator, as not allowed API calls
changed the internal state of the dictionary.

This code was checked against a real bug, issue #1240.

This is what Redis prints (aborting) when a misuse is detected:

Assertion failed: (iter->fingerprint == dictFingerprint(iter->d)),
function dictReleaseIterator, file dict.c, line 587.
4a005a2
@antirez Properly init/release iterators in zunionInterGenericCommand().
This commit does mainly two things:

1) It fixes zunionInterGenericCommand() by removing mass-initialization
of all the iterators used, so that we don't violate the unsafe iterator
API of dictionaries. This fixes issue #1240.

2) Since the zui* APIs required the allocator to be initialized in the
zsetopsrc structure in order to use non-iterator related APIs, this
commit fixes this strict requirement by accessing objects directly via
the op->subject->ptr pointer we have to the object.
bf3932e
@antirez Fix comments for correctness in zunionInterGenericCommand().
Related to issue #1240.
d9bf918
@antirez dictFingerprint() fingerprinting made more robust.
The previous hashing used the trivial algorithm of xoring the integers
together. This is not optimal as it is very likely that different
hash table setups will hash the same, for instance an hash table at the
start of the rehashing process, and at the end, will have the same
fingerprint.

Now we hash N integers in a smarter way, by summing every integer to the
previous hash, and taking the integer hashing again (see the code for
further details). This way it is a lot less likely that we get a
collision. Moreover this way of hashing explicitly protects from the
same set of integers in a different order to hash to the same number.

This commit is related to issue #1240.
3a40b02
@antirez Added redisassert.h as drop in replacement for assert.h.
By using redisassert.h version of assert() you get stack traces in the
log instead of a process disappearing on assertions.
85070a3
@antirez assert.h replaced with redisassert.h when appropriate.
Also a warning was suppressed by including unistd.h in redisassert.h
(needed for _exit()).
e16dd1d
@antirez Fixed type in dict.c comment: 265 -> 256. 19de8e4
@antirez Revert "Fixed type in dict.c comment: 265 -> 256."
This reverts commit 19de8e4.
3bacf4c
@antirez dictFingerprint(): cast pointers to integer of same size. 718fd3c
@antirez Use printf %zu specifier to print private_dirty. e03c730
@antirez Redis 2.6.15 2bfa679
@antirez Update server.lastbgsave_status when fork() fails. 7bf2690
@antirez DEBUG SDSLEN added.
This command is only useful for low-level debugging of memory issues due
to sds wasting memory as empty buffer at the end of the string.
e5fe66e
@antirez Don't over-allocate the sds string for large bulk requests.
The call to sdsMakeRoomFor() did not accounted for the amount of data
already present in the query buffer, resulting into over-allocation.
84472a3
@antirez Fix DEBUG SDSLEN after 2.8 back port. 6b61e94
@antirez tryObjectEncoding(): don't call stringl2() for too big strings.
We are sure that a string that is longer than 21 chars cannot be
represented by a 64 bit signed integer, as -(2^64) is 21 chars:

strlen(-18446744073709551616) => 21
b5cc22d
@antirez tryObjectEncoding(): optimize sds strings if possible.
When no encoding is possible, at least try to reallocate the sds string
with one that does not waste memory (with free space at the end of the
buffer) when the string is large enough.
e3cbc01
@antirez Fix an hypothetical issue in processMultibulkBuffer(). ef0fa08
@yihuang @antirez yihuang fix lua_cmsgpack pack map as array 477e4da
@antirez Redis 2.6.16. bf8d0e6
@antirez redis-cli: fix big keys search when the key no longer exist.
The code freed a reply object that was never created, resulting in a
segfault every time randomkey returned a key that was deleted before we
queried it for size.
1fc83e6
@antirez Allow AUTH / PING when disconnected from slave and serve-stale-data i…
…s no.
f3a7140
Jonah H. Harris Added Intel SIMD Bitwise Operation Support c317afc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment