Github should not elide simple secret values like "1", "0" or "yes", "no" etc. (and maybe it shouldn't do it at all for short strings!)

[demerphq]

Select Topic Area

Bug

Body

Github elides secrets from the output of CI runs, such as tests and what not. This includes very simple values like '1' and '0' and short sequences of numeric and alphabetic data.

This is actually problematic from a security point of view, given sufficient time the secret values can be exposed by having test output produce large numbers of strings with different permutations of characters, if the output is elided you know the value is in the secrets. Agreed you can argue that doing so for a non-trivial length value would be prohibitively time consuming and would require many fetches that might constitute a different class of attack, but the fact remains that eliding the values provides an attacker proof that they have found a secret. Consider if i want to check if a word in a dictionary is a value in the secrets then all i have to is get a test to output the contents fo the dictionary and then check to see which words are elided.

Regardless of your position on this above issue, I think that the eliding of very simple values, such as "1", "0", (and likely "yes", "no" or similar), is very unhelpful to development and does not improve security at all. Consider the CI run here:

https://github.com/Perl/perl5/actions/runs/4238660267/jobs/7365962545#step:10:2648

The output looks like this:

../lib/B/Deparse.t ................................................... ok

#   Failed test 'expecting $count < (***0_000 * ***)'
#   at ../lib/Benchmark.t line 90.
#     '***000***'
#         <
#     '***0000'
Either this system is extremely fast, or the time() function
is broken.

I happen to know that the rightmost elided value in '# Failed test 'expecting $count < (***0_000 * ***)'' is 1. Eliding a value like 1 is very unhelpful. What exactly were all the numbers in the above data? Likely the output should have read:

#     '10001'
#         <
#     '10000'

I do not see how eliding '1' helps secure anything, but it definitely does make it harder to understand the CI output.

Also, short values are especially vulnerable to the dictionary attack I mentioned above. Eliding anything with a modest number of digits is not securing anything, it actually helps leak the secure data. For instance, if I want to find out what (if any) 3 digits values are stored in the secret file all i have to do is write a test that outputs

Details

000001002003004005006007008009010011012013014015016017018019020021022023024025026027028029030031032033034035036037038039
040041042043044045046047048049050051052053054055056057058059060061062063064065066067068069070071072073074075076077078079
080081082083084085086087088089090091092093094095096097098099100101102103104105106107108109110111112113114115116117118119
120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159
160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199
200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239
240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279
280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319
320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359
360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399
400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439
440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479
480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519
520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559
560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599
600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639
640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679
680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719
720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759
760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799
800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839
840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879
880881882883884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919
920921922923924925926927928929930931932933934935936937938939940941942943944945946947948949950951952953954955956957958959
960961962963964965966967968969970971972973974975976977978979980981982983984985986987988989990991992993994995996997998999

and then check to see which digits were elided. Same concept for 4 digits, etc.

Eliding values like this is an example of "security theater", it does not secure the data, and in fact is worse than "security theater" as it makes the secrets less secure by providing an oracle that supports bulk validation checks that I doubt github is in the position to monitor for. All it does is make development more difficult.

Please stop eliding short digit strings, especially single digit or character digit strings.