Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1 Second TTL Expires (Nearly) Immediately #307

Open
neoform opened this issue Oct 24, 2017 · 12 comments

Comments

Projects
None yet
5 participants
@neoform
Copy link

commented Oct 24, 2017

When you set the TTL for an item when the clock is nearing the end of that second (eg, 21:00:00.999999) the item expires 0.000001 seconds later instead of 1 second later.

@neoform neoform changed the title 1 Second TTL Expires Immediately 1 Second TTL Expires (Nearly) Immediately Oct 24, 2017

@dormando

This comment has been minimized.

Copy link
Member

commented Oct 25, 2017

Sadly since there's no high resolution clock in use, that's a thing that'll happen. The smallest realistic TTL is 2, and that'll still be +/- 1.

It's always been this way though, and not sure it should be changed. Should I document this better somewhere before closing the issue?

@neoform

This comment has been minimized.

Copy link
Author

commented Oct 25, 2017

I think it could be helpful to others in the future if it was clearly stated somewhere. We've been getting burned by this for over a month and took is quite a while to narrow it down to this issue. We use MC as the locking mechanism in a mutex class on PH (using the ADD command), which in some cases we want them to have a very short TTL, but weren't aware that we'd be losing up to a full second from it. ;)

@nitrix

This comment has been minimized.

Copy link

commented Oct 25, 2017

It could be worth replacing the comparison that's done from <= to <. It'd effectively make all the keys live a second longer, but also allow for people to use small TTLs (like 1 second) and correctly obtain a least a whole second of liveness.

In my opinion, it's better to have the keys live a second too long rather than a second too short. As long as there are no evictions, TTLs should behave like contracts in that manner.

If it stays how it is, then I think it should be documented to avoid other people with real-time applications running into the same issue.

@Moustachauve

This comment has been minimized.

Copy link

commented Oct 25, 2017

Maybe a one second TTL should be refused?

@dormando

This comment has been minimized.

Copy link
Member

commented Oct 25, 2017

Sadly, someone somewhere relies on the existing behavior. It's kind of a nightmare to change, and I've already had a lot of bloat with starttime twiddles.

I can document it in doc/protocol.txt, and maybe parts of the wiki, but it's hard to guarantee people will even see that :)

@neoform

This comment has been minimized.

Copy link
Author

commented Oct 25, 2017

It might save someone else a bit of trouble. It took us a pretty long time to narrow it down, since it wasn't really mentioned anywhere online (that we could find). We didn't think that this could be the source of our problem; we figured if this was the case, someone else would have had it happen to them given the maturity of MC.

@dormando

This comment has been minimized.

Copy link
Member

commented Oct 25, 2017

I'm sure some folks have, but few reach out, so thanks :) I'll make sure it's added to any documentation discussing TTL's before closing the issue here.

@talis2

This comment has been minimized.

Copy link

commented Nov 1, 2017

Nitrix suggest to replace =< with < is hands down MOST logical.. it CANNOT affect anyones code (according to dormando "existing behavior" ) .. as timing accuracy is already +/- 1sec. Thus < cannot possibly change ANY code except guarantee a 1 sec TTL (range 1.000001 - 1.999999) that is still within a +/-1Sec timeframe ( 1sec + 1sec = 2 ) plus 1 microsec in worst case. Neoform team ought not to have experienced that downtime .. (red faces). May i ask for links to the "bloat" start-time twiddle issues ?

Talks with Mellanox prove one thing .. to offload compute or to bypass kernel on a in-memory store is an only way to gain bandwidth .. Why bog down a distributed access with old protocol ? if i may quote .. " InfiniBand and RoCE employ lossless link-level flow control, namely, credit-based flow control and Priority Flow Control. Even with unreliable transports (UC/UD), packets are never lost due to buffer overflows. If i store all my data in DRAM then i have 100% mem-cache hit , the only worry is BER ..
Where is CRC to be found in the request / response headers ? Is that a most obvious scary thing ? even tho checksum is weak.. is this done on the NIC ? ( IB does this on the adapter, 8 / 10 or 64/66 basis)
RoCE is great but IB is better ..cannot find any meta solution to a straightforward all IB (internal) cluster connect. Lastly, why do i need 300uS latency when i enjoy 3 uS on Infiniband ? .. thats 2 orders of mag @# may we have some benchmarks, as per https://hazelcast.com/resources/benchmark-memcached/ .. or compared w/ REDIS .. anyone seen a REDIS solution that has no mem leak ?

@dormando

This comment has been minimized.

Copy link
Member

commented Nov 2, 2017

It affects code if they set items with a 1s TTL and expect them to be missing some of the time; ie a probabilistic rate limit or lock. Though it's a guess; the truth is anything exposed by an API to the public ends up being used by someone.

I'm not clear on what the rest of your comment is asking about. I've been working with mellanox cards for over a decade and the only thing consistent with them is their production performance is typically 25-50% of what marketing promises. They make a bulk of their sales by undercutting intel on price and aggressively finding customers until someone buys without doing careful tests first (or they don't actually need the performance but enjoy being promised it). I'd take anything they tell you with a grain of salt.

@talis2

This comment has been minimized.

Copy link

commented Nov 2, 2017

@talis2

This comment has been minimized.

Copy link

commented Nov 3, 2017

Can this run on : Argo node OS, specializes a single kernel into multiple aspects that provide:
– Lean OS environments for various OS-bypass needs and next generation HPC runtime support
– Fully-fledged Linux environment:
• Node booting
• complex resource management
• Bulk resource allocation
• Legacy application execution
The specialization is fulfilled over Linux kernel with cgroups, resource controllers/ new kernel additions

Prior referred spec may be +0/-1 Sec (did not look at code ; ) BTW , the compute is not RoCE driven (horrible waste of resources, for local cluster in-mem computation) RoCE is only useful to funnel data from remote server locations (country-wide) The future looks bright for applications that return statistics on data without moving the data, thus in our situation, as the data is posted (by mobile client App sensors) we have luxury in a kNN background compute (on cluster DRAM) to selectively pin the results and modify related metadata to assist a public access info-base to return vital stats (8 packets) rather than violin strokin sales brag .. Is there possibility to add atomic operators to MemCache on a custom kernel call .. it is that potential i wish to explore .. for linear performance gains on incremental cluster size 8 , 16 , 32 nodes ... have 2 x 16 node avail (tho 8 node with E 26x0 is 128 cores) like to start there, as a practical in-mem data size of 512GB ) its either this or REDIS .. there is no other in-memory to handle >200,000 requests /Sec with atomic ops ? Maybe AeroSp. Other MemCache Q's .. i notice expiry as a feature in N seconds .. When =0 (is all timer overhead disabled for fully persistent data) ? Stop daemon, for all unused, without tears? does persistence introduce potential leakage ?

i only store data tables : "..you'll still want to have the cache expire occasionally. In case your app has a bug, a crash, a network blip, or some other issue where the cache could become out of sync." the only blip comes from MemCache ..as in mem leak ..suppose that is likely with large client number ?

what is its codebase? Can it run LUA , what safeguards needed for ASM ? Any boolean ops in future ?

Then theres ESI .. the only dynamics i need is for meta-data (as it changes) so ESI is MemCache feature?
All rendering is preset at the client mobile App, only need meta-data (and in some cases data overlays)
There will be no SQL .. the DB only needs structure, thats provided by external code, it seems, from this

**_Clients are not able to store all types of complex structures. You must turn the data into a pure array, or hash/table type structure before being able to store or retrieve it. ( Externally ? how is the Array sorted .. Pls links to elucidate multi-dimensional storage / GET .. Link an entire book on the subject.

Since item flags are used when storing the item, the same client is able to know whether or not to deserialize a value before returning it on a 'get'. You don't have to do anything special._** More , pls.

i dont wish this to occur (as common with wrapper offload nightmares) .. (solved by removing handlers)
" These handlers are not removed; therefore memory usage will increase every time a request is received. Not only that, but every time the redis client receives a message, the function function(channel, message) is run. Presumably it doesn't do anything if res has already been sent, but it may be that this would incur a significant overhead if there were many handlers " Auto removal is best, based on Client inactivity. Is that available on say 5 Sec timeout? Anything can be coded in..

Meanwhile, i agree that aggressive marketing needs to be put in a dumpster. Mell are hypocritic in still selling Connect X2 VPI cards , while dropping all support from WIN OFED 4.9 (not that i wish to run on windows) when X2 VPI are perfectly suited for IB , not SR-IOV (who needs to virtualize a PCIe bottleneck or Server Sync timestamp (only for FOREX traders, who can afford Mlx-6) ..

https://community.mellanox.com/message/4473#comment-4473

meanwhile RoCE still has its problems .. who says RoCE is the only RDMA possible?

https://community.mellanox.com/thread/3940

@dormando

This comment has been minimized.

Copy link
Member

commented Nov 3, 2017

That's outside the scope of this project, and this issue. I'll fix the documentation when I get a chance.

@memcached memcached locked and limited conversation to collaborators Nov 3, 2017

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.