Pipal lag and failure to complete analysis (RAM limitations?)

[tokyoneon]

Hey @digininja , I would like to feature pipal on Null Byte, but I'm experiencing an issue.

The wordlist being analyzed is 92,400,000 lines, 1.1Gb. Pipal seems to lag after ~2 hours of processing. At that point my CPU usage drops down to normal levels and the ETA starts to slowly increase until it stops. This was done in a Kali VM with 4 cores and 6Gb RAM. Tried again on a dedicated Ubuntu machine with an i7 and 16Gb RAM -- same issue.

Does pipal have limitations? What might be causing the lag and how can I debug?

EDIT:

I just realized Ruby v1.9 is recommended in the README.

> apt-cache policy ruby
ruby:
  Installed: 1:2.5.0
  Candidate: 1:2.5.0
  Version table:
 *** 1:2.5.0 500
	500 http://http.kali.org/kali kali-rolling/main amd64 Packages
	100 /var/lib/dpkg/status

But pipal works fine when analyzing smaller wordlists using v1.2. Should I try manually installing v1.9.x in Kali? There's an APT candidate for v2.3 and v2.5, will those suffice?

P.S. I'm not a ruby coder

[digininja]

The readme is out of date, 1.9 was recommended over 1.87. There is an issue with memory usage as it caches the results as it goes through so with such a big list it will be eating a lot of memory. I've got a rewrite that isn't release yet that stores the results on disk rather than in memory and runs quite a bit faster. I should try to get the finished off and released. Till then, try using just one checker at a time, that will reduce the memory usage.

…

On Sat, 7 Apr 2018, 22:30 tokyoneon, ***@***.***> wrote: Hey @digininja <https://github.com/digininja> , I would like to feature pipal on Null Byte <https://creator.wonderhowto.com/tokyoneon/>, but I'm experiencing an issue. The wordlist being analyzed is 92,400,000 lines, 1.1Gb. Pipal seems to lag after ~2 hours of processing. At that point my CPU usage drops down to normal levels and the ETA starts to slowly increase until it stops. This was done in a Kali VM with 4 cores and 6Gb RAM. Tried again on a dedicated Ubuntu machine with an i7 and 16Gb RAM -- same issue. Does pipal have limitations? What might be causing the lag and how can I debug? EDIT: I just realized Ruby v1.9 is recommended in the README. > apt-cache policy ruby ruby: Installed: 1:2.5.0 Candidate: 1:2.5.0 Version table: *** 1:2.5.0 500 500 http://http.kali.org/kali kali-rolling/main amd64 Packages 100 /var/lib/dpkg/status But pipal works fine when analyzing smaller wordlists using v1.2. Should I try manually installing v1.9.x in Kali? There's an APT candidate for v2.3 and v2.5, will those suffice? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHJWYZUOH6RgLuXVwsmx62TuEV3sWqdks5tmS_ngaJpZM4TLQt7> .

[tokyoneon]

Thanks for the reply. I think the basic.rb checker is the only one enabled.

Do you have an estimated time frame for when the rewrite will be uploaded to GitHub? I wanted to publish my article ~Tuesday but I might be able to postpone it a few days. I would really like to analyze this large wordlist and feature it in the article.

[digininja]

It won't be this week. Would you be able to send me the list? I'll run it through and send you the results, would be a good test.

…

On Sat, 7 Apr 2018, 22:52 tokyoneon, ***@***.***> wrote: Thanks for the reply. I think the basic.rb checker is the only one enabled. Do you have an estimated time frame for when the rewrite will be uploaded to GitHub? I wanted to publish my article ~Tuesday but I might be able to postpone it a few days. I would really like to analyze this large wordlist and feature it in the article. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHJWR-nTAKCv_efubB9f7tNdjbnUemPks5tmTUSgaJpZM4TLQt7> .

[tokyoneon]

-----BEGIN PGP MESSAGE-----
Version: GnuPG v2

hQEMA3E/TAUfwjjMAQf/cVlnFK8SwdHmrKPkDYnpNveAxO2hSCvo4xlcYpl7DMUL
wwrZZn4r7gFmz6L9Gex3kMrVlKRnyxH1hfsAVMBhy2pIi8AUq+GK8pF2Qx1h4dxO
inBIccCUQJUh1qMRDsYg5/IAH2Zih58qjvFZCwA7fczEenbBmrPtZA0V1WgwJnHY
XmLo3w7xbq1ivkSoMxFc733oFY1ez9etS8jsgKwlvQoJaeGJHmfFhVRLkVseAYg/
MhKHWKwn9IWRq6vCrz5HXh35Rdx3wNY6UIF7R7lgEuK3jlJoI3W+8QCOwGLcqdyk
yBKD+A68GVtngNRAx2dt1Zx+WKhmY98+zVcKUYUjGIUCDAOFGPumG2fL5gEP/0E4
xM7w6yyNv4MVXXUFfdydCti15L9D4MaZ6SEiUnYkrscj70uCxXWFSMPNwOXYOrir
NeNSwP2j9NgqDy1270q3D0vOG8N6N1MXQeta8nKRm2nAW7qRjVOqnv20fdv4jkue
NPN1Yx3d6S0sKLESnDRJOPNQLppO/OV4hSND4Rr1reslJ0eBE7fmsRM0qsXngc1V
xuBBsrJg3qdNZjTJ4N/IgcHEaKrlt1QdsRKVmt8CZveFYqvvb7zblfEcjnUh3+95
VVhdSX9cl4M7XNVjWwXGqtgLvCpjhx+YgMVLdAOG0aUY7m4nEL5uh5WlODmzT86F
ZRjK/8tgTNpFG34uJudfEmj4SQrEMiH5ul/fcJjh59HqhlBb09UBsk8m5fiZWL8D
MUJD5xFPBblm7p7lDQ3JOaXabROG5FvYRemS34vFLBnH8eLJRgS4ok/GUG+NrSbv
VE8nELxTUt5pWQNafCG7CyCPaiuFvEOjj70tpQV4/GodUf7nKcezYetsk4H/POaw
mgK39MYpC+nMl7hB4skjWGkcIX4OodpTcGW6sDv+ix+NhT9GnfgCr0u/4hYaB5CC
wDUjpPnAQ+4NfXrq567RZUdLj8FNmyiDDaSad+PDRsnGU/LcwsXLEu54BO11F3qX
E4VQFf8pJvpBG/Q3gW5gQF15G2os+fqxcsjg7A7b0ukB6XgMaRqYM4HQKdUjngfZ
L59+jSr4GrJ6KUSzFApvzNxpBA97aqpyDhiUBBAAGhPqOxRuNqUJnk2/2Ch5TEZa
Tf+QHeGwPS+aKU1Wle3ZnfK/DP0lhES9zpesl/PMAxRrRi46ocq3E0rDYnSneAfr
duO+f92MoBh56uvdhsGkkaub5Qa8OTO3XO42txZvvx52QyvQgZuXRTar8dbijqlX
LoLdxSP5QzdIDmJxG0/oZ8zsPtry+vzULcbeoWK6LJPZunolUl9EZ9178re28H20
Sh/f0wDQWPBFzW/NclEb1zOfKkfW7D4uujUBeZSsqyIHOut/DJjJYlVaqdb8KAYN
Crma6TTkLXBryzcY6unzhHPg/6Sy1BrZqs9L8LAWPNfUY+ltuzfn23OypmYj7i1z
aOgZzjiwyiP457AI37NOKkWUqNeNS4WOZSxrzPcQHcOnNUPl6u1lP+N/gneI/90K
lcu/F/P9NYT/ItDU4AR294aq3JmR40pMh50KFM/JzeApymA6zwLcj8gx7H2NqTxd
eBwq7ERyzXFOpnsfvJjXeV1yWqv0ZrSXkZX1ZwjmI0zi1A4selZXzNWHbnJuJGQD
UdyfjryZNxelcjYagY9kWVYpCJxU8ZL3Dp8hpbHsepNNZ0l1Ubf1Eq8+zpijgjmg
jVwdlidNdimnc4xXsbb+1TzgsaHmXau3oeY2uqIST9W6hXIpMLk1CJV1Xkx8hqfc
2J+KD2zRcuORdWBk/jYR0fQw6wNdIGRBC92c37c=
=dbMz
-----END PGP MESSAGE-----

[tokyoneon]

Thanks for your help btw. Think you can analyze it this weekend?

[digininja]

Unfortunately I'm currently sat in a hotel about to run a marathon so might not be home in time tonight to do it. Where did you get the PGP key? If it was keybase I can read your message, if not, try again and I'll go out and try to kill the key wherever it is.

…

On Sun, 8 Apr 2018, 02:59 tokyoneon, ***@***.***> wrote: Reopened #42 <#42>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHJWe2JOdvM8-X72SfROeiAkdGi_I6aks5tmW72gaJpZM4TLQt7> .

[tokyoneon]

Yup, Keybase. I did notice an older key on the MIT database, but figured it was revoked/expired.

    > gpg2 --fingerprint digininja
pub   rsa4096/E447F6DB 2015-03-02 [SCEA]
      Key fingerprint = 79D0 248B 504B 5B30 6911  7DEF D671 D402 E447 F6DB
uid         [ unknown] keybase.io/digininja <digininja@keybase.io>
sub   rsa2048/6236EB96 2015-03-02 [S] [expires: 2023-02-28]
sub   rsa2048/1FC238CC 2015-03-02 [E] [expires: 2023-02-28]

I think I'll postpone the article till Thursday if you don't get to analyzing the database this weekend. The article will be more interesting for it. So it's worth the wait 👍

[tokyoneon]

Oh, if possible, get the --top 500 and enable the below checkers?

-rw-r--r-- 1 root root  11K Apr  7 14:56 basic.rb
-rw-r--r-- 1 root root 1.6K Apr  7 14:56 EN_religion_checker.rb
-rw-r--r-- 1 root root 1.2K Apr  7 14:56 EN_military_checker.rb
-rw-r--r-- 1 root root 1.4K Apr  7 14:56 EN_family_checker.rb
-rw-r--r-- 1 root root 1.5K Apr  7 14:56 EN_explicit_checker.rb
-rw-r--r-- 1 root root 1.1K Apr  7 14:56 EN_emotion_checker.rb
-rw-r--r-- 1 root root  510 Apr  7 14:56 EN_colour_checker.rb
-rw-r--r-- 1 root root  757 Apr  7 14:56 EN_violence_checker.rb
-rw-r--r-- 1 root root  758 Apr  7 14:56 EN_vehicle_checker.rb
-rw-r--r-- 1 root root  825 Apr  7 14:56 EN_sport_checker.rb
-rw-r--r-- 1 root root 1000 Apr  7 14:56 special_checker.rb

[tokyoneon]

Hey, really sorry, looks like I deleted the keybase app from my other computer. I can't view the messages you sent me. I would have to reset my entire Keybase account to regain access Keybase's chat functionality, but doing so would still wipe my previous message history.

Would you be able to send me PGP blocks here or via email? Really sorry for the inconvenience. #EncryptionFail

[digininja]

Ye sure, will send it over in some way. Not all the checkers are working yet but I'll get as many going as I can.

…

On Mon, 9 Apr 2018, 02:20 tokyoneon, ***@***.***> wrote: Hey, really sorry, looks like I deleted the keybase app from my other computer. I can't view the messages you sent me. I would have to reset my entire Keybase account to regain access Keybase's chat functionality, but doing so would still wipe my previous message history. Would you be able to send me PGP blocks here or via email? Really sorry for the inconvenience. #EncryptionFail — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#42 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHJWSXQF0ZAqgIy1sgPEJUyLh_6tv1Wks5tmrdwgaJpZM4TLQt7> .