Keyboard locks and outputs repeating sequence of same key about 100 times

[blais]

This bug occurs rarely maybe a few times per day or so.

The keyboards appears to lock up - typing keys won't outtput them - and either just ssits frozen or repeats the last key pressed about 100 times over 2-3 seconds. Then it resumes as normal.

[blais]

Actually, no, this happens much more often, I want to say every 3-5 minutes. Most of the time the lock up does not output a repeated key, just sometimes.

[blais]

FYI this might be related to the DebounceThrottleDiv, I've got it set to 20 at the moment.

[haata]

So, using 20 for DebounceThrottleDiv is a 2^20. That's one scan every 1 048 576 loops, which is a little excessive.
On a 48 MHz processor that's only 45.8 scans per second if there was nothing else going on. USB and the kll macro engine also have stuff to do in the background (not to mention if the cli is being used).

Thanks for testing this though. Something like 2^15 would probably be much more reasonable.

[blais]

Thanks for the feeedback;
I will reset it to use 0xFFFF and 2^15 which is about 1500 scaneees/s:

mandarine:~$ dc -e "1 k 48 1000 1000 * * p 2 15 ^ p / p"
48000000
32768
1464.8

On Sat, Mar 7, 2015 at 1:49 PM, Jacob Alexander notifications@github.com
wrote:

So, using 20 for DebounceThrottleDiv is a 2^20. That's one scan every 1
048 576 loops, which is a little excessive.
On a 48 MHz processor that's only 45.8 scans per second if there was
nothing else going on. USB and the kll macro engine also have stuff to do
in the background (not to mention if the cli is being used).

Thanks for testing this though. Something like 2^15 would probably be much
more reasonable.

—
Reply to this email directly or view it on GitHub
#23 (comment).

[blais]

Just tested... appears to be worse:
m m m m m m m mmm m m m m m m mm m mm m m m m m m m m m m m mm m m
m m m m m mm m m m m m m mm m m m m mm m m m m m m mm mm m m m m m
mm mm m m m m m mm mm

(I wish I had time to hack on tthis, but I'm buried by my open source
project and have to figure out a complex wash sales calculation situation.
The govt wins.)

On Sat, Mar 7, 2015 at 7:05 PM, Martin Blais blais@furius.ca wrote:

Thanks for the feeedback;
I will reset it to use 0xFFFF and 2^15 which is about 1500 scaneees/s:

mandarine:~$ dc -e "1 k 48 1000 1000 * * p 2 15 ^ p / p"
48000000
32768
1464.8

On Sat, Mar 7, 2015 at 1:49 PM, Jacob Alexander notifications@github.com
wrote:

So, using 20 for DebounceThrottleDiv is a 2^20. That's one scan every 1
048 576 loops, which is a little excessive.
On a 48 MHz processor that's only 45.8 scans per second if there was
nothing else going on. USB and the kll macro engine also have stuff to do
in the background (not to mention if the cli is being used).

Thanks for testing this though. Something like 2^15 would probably be
much more reasonable.

—
Reply to this email directly or view it on GitHub
#23 (comment).

[blais]

Here's theee right way to solve this:
accumulate statisticss aboutt buiitton press times, look at the
disttribution, select the right cutoff point and set the timings
accordingly. I think the keyboard couuld actually self-calibrate to avoid
reepeats in order to simplify the computation.

Furthermoreeee, these types of increeedibly fast rrepeeats should be
deetectable and preeventable. In addition to dealing with bounces directly,
we could insert code tthat prevents two key signals in the output within a
very short delay.

On Sat, Mar 7, 2015 at 7:07 PM, Martin Blais blais@furius.ca wrote:

Just tested... appears to be worse:
m m m m m m m mmm m m m m m m mm m mm m m m m m m m m m m m mm m m
m m m m m mm m m m m m m mm m m m m mm m m m m m m mm mm m m m m m
mm mm m m m m m mm mm

(I wish I had time to hack on tthis, but I'm buried by my open source
project and have to figure out a complex wash sales calculation situation.
The govt wins.)

On Sat, Mar 7, 2015 at 7:05 PM, Martin Blais blais@furius.ca wrote:

Thanks for the feeedback;
I will reset it to use 0xFFFF and 2^15 which is about 1500 scaneees/s:

mandarine:~$ dc -e "1 k 48 1000 1000 * * p 2 15 ^ p / p"
48000000
32768
1464.8

On Sat, Mar 7, 2015 at 1:49 PM, Jacob Alexander <notifications@github.com

wrote:

So, using 20 for DebounceThrottleDiv is a 2^20. That's one scan every 1
048 576 loops, which is a little excessive.
On a 48 MHz processor that's only 45.8 scans per second if there was
nothing else going on. USB and the kll macro engine also have stuff to do
in the background (not to mention if the cli is being used).

Thanks for testing this though. Something like 2^15 would probably be
much more reasonable.

—
Reply to this email directly or view it on GitHub
#23 (comment).

[haata]

So, I'm interested, would you mind taking a picture of the soldering? (I'm
quite drunk on bourbon right now, so humour me :P)

The algorithm itself should be quite solid, so I'm curious to what the heck
is going on here that's different from all the other keyboards I've seen.
Also, what's your WPM?

On Sat, Mar 7, 2015 at 4:11 PM, Martin Blais notifications@github.com
wrote:

Here's theee right way to solve this:
accumulate statisticss aboutt buiitton press times, look at the
disttribution, select the right cutoff point and set the timings
accordingly. I think the keyboard couuld actually self-calibrate to avoid
reepeats in order to simplify the computation.

Furthermoreeee, these types of increeedibly fast rrepeeats should be
deetectable and preeventable. In addition to dealing with bounces directly,
we could insert code tthat prevents two key signals in the output within a
very short delay.

On Sat, Mar 7, 2015 at 7:07 PM, Martin Blais blais@furius.ca wrote:

Just tested... appears to be worse:
m m m m m m m mmm m m m m m m mm m mm m m m m m m m m m m m mm m m
m m m m m mm m m m m m m mm m m m m mm m m m m m m mm mm m m m m m
mm mm m m m m m mm mm

(I wish I had time to hack on tthis, but I'm buried by my open source
project and have to figure out a complex wash sales calculation
situation.
The govt wins.)

On Sat, Mar 7, 2015 at 7:05 PM, Martin Blais blais@furius.ca wrote:

Thanks for the feeedback;
I will reset it to use 0xFFFF and 2^15 which is about 1500 scaneees/s:

mandarine:~$ dc -e "1 k 48 1000 1000 * * p 2 15 ^ p / p"
48000000
32768
1464.8

On Sat, Mar 7, 2015 at 1:49 PM, Jacob Alexander <
notifications@github.com

wrote:

So, using 20 for DebounceThrottleDiv is a 2^20. That's one scan every 1
048 576 loops, which is a little excessive.
On a 48 MHz processor that's only 45.8 scans per second if there was
nothing else going on. USB and the kll macro engine also have stuff to
do
in the background (not to mention if the cli is being used).

Thanks for testing this though. Something like 2^15 would probably be
much more reasonable.

—
Reply to this email directly or view it on GitHub
<#23 (comment)
.

—
Reply to this email directly or view it on GitHub
#23 (comment).

[haata]

So, I just assembled another infinity keyboard. And I noticed it had some bouncy switches (Matias Linears).

I didn't use my normal soldering iron (MFR-2200) and did it rather quickly.
Before trying to debug the debouncing too much I went over the problem switches again with a soldering iron (making sure the solder completely melted).
So far, my debouncing issues haven't come up again (I'm using the stock debounce settings).

[blais]

I have no idea how to tell if it's off.
It's definitely not perfect, but I think all the contacts are sufficient.
How do I tell?
Can you see from this?
http://furius.ca/tmp/infinity/

I can make a higher res photo later on.

I'll check if the repeating is switch-specific by doing some tests, if
that's the case, that would be telling.

On Sun, Mar 8, 2015 at 4:53 PM, Jacob Alexander notifications@github.com
wrote:

So, I just assembled another infinity keyboard. And I noticed it had some
bouncy switches (Matias Linears).

I didn't use my normal soldering iron (MFR-2200) and did it rather quickly.
Before trying to debug the debouncing too much I went over the problem
switches again with a soldering iron (making sure the solder completely
melted).
So far, my debouncing issues haven't come up again (I'm using the stock
debounce settings).

—
Reply to this email directly or view it on GitHub
#23 (comment).

[haata]

Yeah, I'm definitely not just trying to get out of fixing this in code :package :P

Visually, it's often pretty hard to tell. Sometimes you can if the soldering is extremely bad. Yours isn't in that category :)
A little experiment you can try though. Find a switch that bounces pretty badly, take a soldering iron and re-heat the switch pins for a few seconds after the solder has melted entirely. This way, you're absolutely sure that the contact is good.
You might also want to try heating up the solder joints of the diode as well (less likely to be the issue) for that switch.
Then, using the stock configuration, try to see if the switch bounces.

A bad joint may have higher than expected resistance which may cause miss-reads or slow down the current propagation.

[blais]

Alright, more observations after fiddling a fair bit:
It is true that the repeats tend to occur more often on particular keys.
My keyboard has had unwanted repeats mostly on "m", "r", "e", "SPC" and a
few more (it's not obvious to figure out just by observing it). I might
want to redo those joints. I'm not sure how safe that is since it'll
oxidize the solder by doing so (is this really a concern). I'll try using a
generous amount of flow and some more solder on those keys.

However, and most importantly, I have fixed the problem with software.
Using simple code to compute the histogram of time differences between
events (a new "matrixHist" command to print out the results), I was able to
determine the timing of repeated events when they occur and apply a
threshold on them. There was a clear empty gap in the histogram between the
offending events and regular keyboards events, even if you try to type
really fast, so they're detectable. I'm running this version now and my
keyboard appears to work flawlessly (it's such an incredible relief!). Even
the occasional lockups appears to be gone (though honestly I've been using
this code for about 15 minutes only, but I think that's quite enough to
have reproduced them). I'm using the default parameters for
DebounceDivThreshold and DebounceThrottleDiv (both unset).

Alright, my change is a kludge, albeit a working one (inspecting the code
will reveal that immediately). I suspect there may be a bug in the way your
debounce algo detects events, what I mean specifically is that there must
exist some series of sense readings that throws off the current algorithm
and generate multiple events. The lockups are likely triggered by the same
sequence of events but somehow the state gets "stuck" in a longer loop than
just generate 2 or 3 characters before it exits (something like 50-80
characters get output). IMHO the correct thing to do is to figure out how
this can happen (I'll have to think about that one for a bit, it's not
obvious to me right away), and to then rewrite the debouncing algorithm to
avoid failing prey to that sequence of events, and then remove my threshold
kludge. In any case, I should likely make the Matrix_repeatThreshold value
configurable, or better: infer it automatically from the histogram after
keys are pressed for a while (the keyboard would be in that way
self-calibrating, it could even be done for each switch independently).

It would be interesting to add statistics regarding how often the debounce
algo (or my threshold) needs to kick in to avoid bounciness on each key.
Each keyboard could output a measure of how bouncy its switches are, and
this could tell you how bad the soldering job is for each key, guiding one
to redo some specific joints :-)

My changes are here:
blais@c02a290

Merge it if desired, I'll be maintaining my branch for a little while. I
think it needs to be polished a bit, but at least I can use the keyboard
now (I'm bringing this beauty back to work tomorrow...).

Cheers,

On Sun, Mar 8, 2015 at 8:02 PM, Jacob Alexander notifications@github.com
wrote:

Yeah, I'm definitely not just trying to get out of fixing this in code
:package :P

Visually, it's often pretty hard to tell. Sometimes you can if the
soldering is extremely bad. Yours isn't in that category :)
A little experiment you can try though. Find a switch that bounces pretty
badly, take a soldering iron and re-heat the switch pins for a few seconds
after the solder has melted entirely. This way, you're absolutely sure that
the contact is good.
You might also want to try heating up the solder joints of the diode as
well (less likely to be the issue) for that switch.
Then, using the stock configuration, try to see if the switch bounces.

A bad joint may have higher than expected resistance which may cause
miss-reads or slow down the current propagation.

—
Reply to this email directly or view it on GitHub
#23 (comment).

[haata]

Finally got a chance to look at your code. Neat idea, I hope it's working out for you :D

So, I did some basic scan rate testing the other day (hadn't done this originally, oops) and I discovered that debounce algorithm scans each key every ~300 us. Using the cherry defined min debounce time of 5 ms (http://cherrycorp.com/product/mx-series/ Technical Data section):

5000 us / 300 us = 16.66667

This means, at minimum I should be scanning ~17 times per transition. My original calculations show a perfect cycle requiring 13 scans for a decision in the algorithm...
And there are cases where the possible number of scans for a decisions is actually less than that.

So, yep I need to rework some stuff.

[blais]

On Mon, Mar 16, 2015 at 1:05 AM, Jacob Alexander notifications@github.com
wrote:

Finally got a chance to look at your code. Neat idea, I hope it's working
out for you :D

Well the keyboard is usable indeed. I'm using it full time at work. Never a
bounce.

So, I did some basic scan rate testing the other day (hadn't done this
originally, oops) and I discovered that debounce algorithm scans each key
every ~300 us. Using the cherry defined min debounce time of 5 ms (
http://cherrycorp.com/product/mx-series/ Technical Data section):

5000 us / 300 us = 16.66667

This means, at minimum I should be scanning ~17 times per transition. My
original calculations show a perfect cycle requiring 13 scans for a
decision in the algorithm...
And there are cases where the possible number of scans for a decisions is
actually less than that.

Do you really need to scan?
Here's what I mean: Does a switch at rest (or held) position ever signal
a transition?
If not, then it should be sufficient to trigger upon the first detected
transition and ignore everything occurring over the following 5ms.

Why wouldn't this work?

So, yep I need to rework some stuff.

—
Reply to this email directly or view it on GitHub
#23 (comment).

[Duckle29]

Any news on this? I'm having some rather bad repeated characters with yours (blais) firmware, I'm just now flashing HaaTa's back to see if it's better.

[blais]

Have you tried the latest? The thing to do is to look at the histogram and
bump your setting up

On Sat, May 9, 2015, 19:51 Mikkel Jeppesen notifications@github.com wrote:

Any news on this? I'm having some rather bad repeated characters with
yours (blais) firmware, I'm just now flashing HaaTa's back to see if it's
better.

—
Reply to this email directly or view it on GitHub
#23 (comment).

[kaansoral]

(I'm very picky in general, had repeated character issues with V60 for example, occurred during slow clicking the Enter, their improved firmware improved things, but only a switch replacement fixed the issue)

I started using the latest state of the controller firmware, I luckily had no keypress issues with the infinity keyboard, and I'm using clears (They seem to be more prone to keypress issues, since the bump makes the leafs bounce), so I can soft-vouch for the latest state of the firmware

It's also a good idea to replace the problematic switch, some switches have rare issues with activation

Offtopic, I'm wondering whether the bootloader ever needs to be updated // whether it affects the functionality in general?

[Duckle29]

@blais I seem to have misplaced my previous folder (so I'm not making a point of placing it somewhere sensible) Which latest one do you mean? The latest from HaaTa or your latest one?

@kaansoral Hmm now that you mention it, it does seem to be t that does it the most, so I will just try and throw a new switch in there. I'm using blues btw. Wondering if I should try my Gateron blue :P Eh maybe later.

[haata]

I'm putting the finishing touches on the backend for the new configurator right now (it's not as full featured as compiling/editing kll files yourself...yet).
Key bouncing is #3 on my todo list right now (after the configurator and bootloader bug fixes for the infinity ergodox).

@kaansoral The bootloader code doesn't affect general functionality. I have some minor updates I've done to the bootloader (not included on any of the shipping IC60 rev 1 or rev 2 of the infinity though) for some text fields in the descriptors. Perhaps if I, or someone else makes a different style of bootloader it might be useful to flash it (e.g. mass storage instead of dfu).

[Duckle29]

@kaansoral Thanks for the tip. Swapping the switch fixed the bounce issue on my T. Hooked it up like this:

with the output there going to my logic analyzer.

This is the bad switch:

And this is a new, good switch:

Pretty signifficant difference! The good switch bounces for ~50uS while the bad one bounces for almost 1ms!

EDIT:
Oh and I think that @blais idea about just ignoring bounce after a detected state change for x ms is a great idea! I'm considering making a quick bounce testing jig for running a lot of keys through.
Or maybe someone could make a custom firmware that would record bounce duration for say 10 seconds of typing, and then type out the results (It's a keyboard afterall :P)

It would also be interesting to see how fast people type (how quickly can a very fast typist get two keypresses in after each other on a switch?)

[kaansoral]

Well, thank you for the analysis, I should upgrade my keyboard debugging skills, I would really appreciate some debugging/graphing advice at this point

I tested my switches with a multimeter before soldering, some of them (5-6) were non-deterministic just after the activation point, which was concerning, they are just not too reliable if you don't bottom out and hang around the activation point for long, but when soldered, they all seem to work without issues, no double clicking at any activation styles I tested

Up until now I only had one really faulty switch, it was the Enter/clear switch on my V60, it activated when you touched it, when you pressed it slowly, it activated twice

[Duckle29]

@kaansoral I used one of these: http://www.ebay.com/itm/New-USB-Logic-Analyzer-Device-Set-USB-Cable-24MHz-8CH-24MHz-for-ARM-FPGA-/111540669744?pt=LH_DefaultDomain_2&hash=item19f8578d30

Which is a clone of the Saleae logic 8: https://www.saleae.com/logic/ and uses their software
Now I would always recommend buying the original, as you really aren't paying the 219$(!) for the hardware, but for the software. But I simply couldn't justify spending 219$ on something I will use 4 times a year.

Also keep in mind that this is a cheap china clone, and it does have some issues. I've noticed that there's crosstalk between channels, but have yet to bother to see if it's internal, or just the cables, but beware!

I also can't select 24MHz sampling rate in the software (only 16). If that's a linux bug or not, I dunno. (Using the 1.1.34 beta, as it's the version I can actually get to work)

[blais]

https://github.com/blais/controller

Type on your [faulty] keyboard a lot.
Connect to it; look at the histogram of bounces with "matrixHist"
Bump up Matrix_repeatThreshold to the largest bucket.

(Ideally this should be adjusted automatically; there's no reason the
keyboard shouldn't be able to automatically adjust itself.)

On Sat, May 9, 2015 at 8:16 PM, Mikkel Jeppesen notifications@github.com
wrote:

@blais https://github.com/blais I seem to have misplaced my previous
folder (so I'm not making a point of placing it somewhere sensible) Which
latest one do you mean? The latest from HaaTa or your latest one?

@kaansoral https://github.com/kaansoral Hmm now that you mention it, it
does seem to be t that does it the most, so I will just try and throw a new
switch in there. I'm using blues btw. Wondering if I should try my Gateron
blue :P Eh maybe later.

—
Reply to this email directly or view it on GitHub
#23 (comment).

[unphased]

I have a key (happens to be my L key) that has started to "degrade" over the last day or so where I've started to use this keyboard quite a bit.

Anyone else ever experience this gradual increase in the tendency to output repeated keystrokes? l had a similar issue with my C key and I allllready used up my one spare keyswitch that I got (Fixed the C key...). Hoping to get some more sent to me, but in the meantime could someone explain which exactly is the place the debounce parameter is that I can tweak?

(BTW the soldering is pretty good on this L key)

[haata]

Hmm, could be the slow corrosion of the metal contacts.

I'm probably going to work on debounce (today/tomorrow). I have some ideas
on how to hopefully resolve this once and for all.

On Sun, Jun 14, 2015 at 2:02 PM Steven Lu notifications@github.com wrote:

I have a key (happens to be my L key) that has started to "degrade" over
the last day or so where I've started to use this keyboard quite a bit.

Anyone else ever experience this gradual increase in the tendency to
output repeated keystrokes? l had a similar issue with my C key and I
allllready used up my one spare keyswitch that I got. Hoping to get some
more sent to me, but in the meantime could someone explain which exactly is
the place the debounce parameter is that I can tweak?

—
Reply to this email directly or view it on GitHub
#23 (comment).

[unphased]

So I did find this

# This defines how often the matrix is scanned                               
# By, default the key matrix is scanned once per macro processing loop       
# For fast uCs and bouncy switches, this can be non-ideal                    
# 0 - Bit-shift of 0                                                         
# 1 - Bit-shift of 1 (i.e. divide by 2)                                      
# 2 - Bit-shift of 2 (i.e. divide by 4)                                      
# 3 - Bit-shift of 3 (i.e. divide by 8)                                      
# etc.                                                                       
# Depending on the architecture, this is either a maximum of 16 or 32        
# Increasing this value will increase switch latency                         
DebounceThrottleDiv => DebounceThrottleDiv_define;                           
DebounceThrottleDiv = 0; # Default                                           
#DebounceThrottleDiv = 2; # /4 divider                                       

This seems like it's set to 0. So I can play with this up till values near 15 or 16?

[unphased]

And of course, increasing this valllue from 0 to 2 makes the probllem significantly worse, now l repeats a lllot more than just once most of the time now. Interestingly, the rest of the keys seem to be functioning fine, welll except for the spacebar up there like you just saw.

Well, now i set it to 10, seems like after It reloads, and I start spamming and alternating L a lot, it seems to enter a regime of repeating it a lot, and then it dies back down and I guess now overall it is back to being more reliable than before. So that almost seems like some sort of self-calibration period that's happening in there.

[haata]

I've added timing code to the debounce algorithm 98f796d

You can use the MinDebounceTime kll define to control how long to wait before allowing another state transition. By default this is set to 5 ms (which is the usual).

For extremely bouncy switches there is still a chance that there could be bouncing issues. But I would have to increase initial keypress latency to fix this (also re-write the algorithm).

So far I've had good results with the code.

Sorry this has taken so long.

[unphased]

Nice, I had set the old debounce number to 17 and it was working fairly well, but there were occasional pauses where it would spit out a repeated character like 12 times. New code seems to work well so far here.

[haata]

I haven't noticed any issues since I added the new code, closing.