Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

thread 'main' panicked at 'Unable to open cache file!: Os { code: 24, kind: Other, message: "Too many open files" }', src/platforms.rs:37:53 #12

Closed
gprime31 opened this issue Jan 3, 2022 · 14 comments
Closed

thread 'main' panicked at 'Unable to open cache file!: Os { code: 24, kind: Other, message: "Too many open files" }', src/platforms.rs:37:53 #12

gprime31 opened this issue Jan 3, 2022 · 14 comments
Assignees
@TheBinitGhimire
Labels
bug Something isn't working

Comments

@gprime31
Copy link

gprime31 commented Jan 3, 2022

thread 'main' panicked at 'Unable to open cache file!: Os { code: 24, kind: Other, message: "Too many open files" }', src/platforms.rs:37:53

when 100+ subs in a file.
@gprime31 gprime31 added the bug Something isn't working label Jan 3, 2022
@TheBinitGhimire
Copy link
Owner

Hello @gprime31,

Can I know what's the command you used while you got this error message? Are you using the project's in-built mechanism to load the file (i.e. using the -f flag), or are you using some sort of technique that depends on the system/shell you are running on?

I couldn't reproduce the issue with the project's in-built file loading mechanism (I tried with files containing hundreds and thousands of hostnames), so please let me know how you landed up on this error message, as that would help me out in understanding this issue well, before coming up with a fix.

Thanks,
@TheBinitGhimire

@gprime31
Copy link
Author

gprime31 commented Jan 3, 2022 •
edited

NtHiM -f subs.txt -o results.txt
Subs.txt contains https://example.com
https://sub.example.com etc no params or anything.
It also seems to hang on subs that don't resolve or take too long, ex Chinese sites like sub.example.cn.

@gprime31
Copy link
Author

gprime31 commented Jan 3, 2022 •
edited

OS is ubuntu and zsh for shell
Thanks
Works great on most lists.

@TheBinitGhimire
Copy link
Owner

Hi there,

Can you please try using the verbose (-v) flag to check for the non-resolving hostnames? By default, non-resolving ones aren't shown in the output, so I would like to know if it really hangs or at least displays an immediate output in verbose mode for such hostnames?

Till then, I'm looking up on your major issue, thanks for providing all the environment details! 😇😇

Thanks,
@TheBinitGhimire

@TheBinitGhimire
Copy link
Owner

Also, can you please try changing the value of system-wide kernel directives for maximum number of open files? But, before doing so, I'd love to know what your current limit is (use cat /proc/sys/fs/file-max for that purpose).

Here's a sample command if you would like to try changing the limit (put limit higher than existing):

sysctl -w fs.file-max=809600

Please let me know if this solves your issue, so that I could work on this issue further!

@gprime31
Copy link
Author

gprime31 commented Jan 3, 2022

2022-01-03_10-58

@gprime31
Copy link
Author

gprime31 commented Jan 3, 2022

I have ran tools that take 100's of subs as stdin and never ran into this problem, so don't really want to change os configs.
I wonder if it's rust problem?
Go and python have no problems parcing that same file all at once.

@gprime31
Copy link
Author

gprime31 commented Jan 3, 2022 •
edited

also the hanging on .cn sites, I let it run over night and it didn't crash this time but it's taking forever.
maybe you can make a timeout flag? --timeout=3
something like that?

@gprime31
Copy link
Author

gprime31 commented Jan 3, 2022

but I used to run this https://github.com/LukaSikic/subzy for subtakeover.
And it takes the same list and works no problem. no crashes or hangs.
but it's old.
and I like yours better :)

@gprime31
Copy link
Author

gprime31 commented Jan 3, 2022 •
edited

oh and for the verbose:
it just says [ERROR] An error occured for [site.com]
so an option to set the max retries and timeout flags should be the fix here
Because it does move on the the next url in the list eventually. but sometimes takes hours to do so.

@TheBinitGhimire
Copy link
Owner

Hi there,

Thanks for providing all of the information, and also thanks for the suggestion on that new ability! I will be working on the two flags you mentioned (i.e. max-retries and timeout).

I am still working on the major issue you are facing, regarding the file opening thing. I will put up any information or fixes here in the comments if I am able to confirm.

Thanks,
@TheBinitGhimire

TheBinitGhimire added a commit that referenced this issue Jan 15, 2022
@TheBinitGhimire
Implemented the functionality for Timeout during connections! (second…
baafec9 
…ary feature request at #12)
TheBinitGhimire added a commit that referenced this issue Jan 21, 2022
@TheBinitGhimire
Implemented a new strategy to handle signatures, leading to huge perf…
803e91a 
…ormance boost! (Complete Resolution to the issue #12)

- Defined implementations on "Definition" and "Response" structs,
- Defined a "_getPlatformName()" method within the "Response" struct,
- Change the workflow of the "_platforms()" function, in order to simply return the signatures definitions,
- PERFORMANCE BOOST: Implemented a "_findPlatformName()" function, which calls the "_platforms()" function just once throughout the runtime, which leads to the signatures file being read just once (previously, it was reading the file every time for every single hostname)!
@TheBinitGhimire
Copy link
Owner

Hello @gprime31,

The new release of NtHiM, i.e. v0.1.4, is published now, which includes both of the things you mentioned here:

  • Main Fix for your "Too many open files" issue, and,
  • Implementation of the timeout (-s or --timeout) flag!

I hope you would test these functionalities in the new version of NtHiM.

Thanks,
@TheBinitGhimire

@gprime31
Copy link
Author

gprime31 commented Jan 21, 2022 •
edited

Works great now!!! Thanks
had to update rust to latest to compile from source, may want to add it to the install notes.
https://www.rust-lang.org/tools/install and run the sh script.

@TheBinitGhimire
Copy link
Owner

Hello @gprime31,

Thank you for your review on the new release! I feel really good to hear that it is working fine now.

Also, thanks for letting me know that you had to update Rust to compile from source! I will surely include that in the installation instructions portion in the documentation for NtHiM, as you have mentioned in your issue #21.

Thanks,
@TheBinitGhimire

@TheBinitGhimire TheBinitGhimire mentioned this issue Jan 21, 2022
Closed
TheBinitGhimire added a commit that referenced this issue Jan 21, 2022
@TheBinitGhimire
Updated the Installation Instructions, #21! (#12 and #19)
8dd6c45 
Thanks to @gprime19 for opening up the issue #21, that led to this update in the documentation!
TheBinitGhimire added a commit that referenced this issue Jan 21, 2022
@TheBinitGhimire
Updated the Installation Instructions, #21! (#12 and #19)
1beecb3 
Thanks to @gprime31 for opening up the issue #21, that led to this update in the documentation!
TheBinitGhimire added a commit that referenced this issue Jan 21, 2022
@TheBinitGhimire
Updated the Installation Instructions, #21! (#12 and #19)
44e3db5 
Thanks to @gprime31 for opening up the issue #21, that led to this update in the documentation!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@TheBinitGhimire TheBinitGhimire

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@TheBinitGhimire @gprime31