Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gospider produces too many files for cat to handle - resulting in error. #442

Closed
frost19k opened this issue Jan 3, 2022 · 2 comments
Closed
Labels
bug Something isn't working

Comments

@frost19k
Copy link
Contributor

frost19k commented Jan 3, 2022

Describe the bug
It may happen that gospider produces way too many files for cat to handle

❯ ./reconftw.sh -d <Target> -s -v
...
...
 Running : Source code scraping subdomain search 
./reconftw.sh: line 519: /usr/bin/cat: Argument list too long
❯ sed -n 519p reconftw.sh | tr -d '\t'
[[ $NUMFILES -gt 0 ]] && cat .tmp/gospider/* | sed '/^.\{2048\}./d' | anew -q .tmp/gospider.txt
❯ ls -A /root/Tools/reconftw/Recon/<Target>/.tmp/gospider | wc -l
888970

To Reproduce
Run reconFTW.sh in any mode that calls sub_scraping().
Error might or might not show, depending on the target.

Expected behavior

❯ ./reconftw.sh -d <Target> -s -v
...
...
 Running : Source code scraping subdomain search 

 XXX new subs (code scraping)

Desktop (please complete the following information):

  • Architecture: x86_64
  • OS: Manjaro
  • Version: Rolling (Currently 21.2.0)

Additional context
Testing a solution that employs find to feed files one at a time.
Will make a PR if it works.

@frost19k frost19k added the bug Something isn't working label Jan 3, 2022
six2dez added a commit that referenced this issue Jan 8, 2022
@six2dez
Copy link
Owner

six2dez commented Jan 8, 2022

Fixed on dev branch 026956a

@six2dez six2dez closed this as completed Jan 8, 2022
@frost19k
Copy link
Contributor Author

frost19k commented Jan 8, 2022

Hey, thanks for taking care of this.
I'm sorry I didn't get around to making a PR. Turns out my CEH voucher is expiring next month & so I've had to abruptly drop almost everything :-P

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants