-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Topology validate fails with error: dictionary changed size during iteration #131
Comments
Attaching the graphml of the topology which caused the error. |
Need to see if this is still a problem. |
@kthare10 is this error popping up in Orchestrator at graph ingestion? Is it possible somehow the model is being modified by one thread while being validated by another? This code is not thread-safe and at least at first glance this is looking to me like a race condition. Simply validating this graph after loading it doesn't create any problems. |
Yes, the error pops up as soon as Orchestrator is validating the incoming create slice request. No other thread is processing the slice at that point. This function is called directly from the Flask end point handler. I'll try to reproduce it and keep you posted.
|
It looks the issue is because Also, in the current CF implementation, when the topology object goes out of scope it is not explicitly deleting the graph from the NetworkXGraphStorage. So this Singleton instance on production must be growing pretty big. It has resulted in some of the operations taking as long as 250 seconds.
|
Topology validate fails with error: dictionary changed size during iteration
Snapshot of the stack trace:
The text was updated successfully, but these errors were encountered: