RuntimeError: Set changed size during iteration in Photon 1.0.7

[thistehneisen]

[!] Progress: 1/1
[~] Level 2: 387 URLs
[!] Progress: 387/387
[~] Level 3: 18078 URLs
[!] Progress: 18078/18078
[~] Level 4: 90143 URLs
[!] Progress: 39750/90143^C
[~] Crawling 0 JavaScript files

Traceback (most recent call last):
  File "photon.py", line 454, in <module>
    for url in external:
RuntimeError: Set changed size during iteration

[thistehneisen]

I must add that this is not persistent, reproducing it requires stopping the process via SIGINT, but on my second try it worked without problems and saved the results. Might be related to lag if the session is via SSH where the Photon.py process is ran on the other end.

[s0md3v]

Hi there,

You did the right by reporting the error your encountered but I am unable to reproduce it.

[+] URLs retrieved from sitemap.xml: 102
[~] Level 1: 103 URLs
[!] Progress: 103/103
[~] Level 2: 21 URLs
[!] Progress: 21/21
[~] Level 3: 440 URLs
[!] Progress: 200/440^C
[~] Crawling 6 JavaScript files
[!] Progress: 6/6
--------------------------------------------------
[+] URLs: 595
[+] Intel: 119
[+] Files: 3
[+] Endpoints: 43
[+] Fuzzable URLs: 0
[+] Custom strings: 0
[+] JavaScript Files: 6
[+] External References: 169
--------------------------------------------------
[!] Total time taken: 1 minutes 14 seconds
[!] Average request time: 0.173756453048 seconds
[+] Results saved in ******* directory

Would it be appropriate if I mark this issue as invalid & close it?

[thistehneisen]

It wouldn’t, upon four tries the issue was 2/4 positive which makes it false positive negative.

[s0md3v]

I am not really sure what is causing the problem, have you tried reproducing it on your local machine?

The code block which throws the error is this one:

    for url in external:
        if 'github.com' in url or 'facebook.com' in url or 'instagram.com' in url or 'youtube.com' in url:
            intel.add(url)

The error suggests that the set we are iterating over gets changed during iteration but as you can see there's no way that can happen.

[cclauss]

You can “freeze” it by changing it to for url in tuple(external):.

[s0md3v]

Yep @cclauss , but do you have any idea why its happening?

[cclauss]

My guess as to why this is happening is that one of more threads is adding (or deleting) to external while another thread is trying to iterate through it. https://docs.python.org/3/library/threading.html contains info on Locks and Semaphores, etc. that can be used to guard against these multithread contention issues.

[thistehneisen]

Can be the case, used 50 threads at the time.

[cclauss]

In such a multi-threaded environment, https://docs.python.org/3/library/queue.html will be much safer than a list.

[s0md3v]

I don't think so, I always use 100 threads and never faced any error.
But I will consider using Queue as suggested by @cclauss if it doesn't affect performance.