Performance improvements #271
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
istalker2
force-pushed
the
performance-improvements
branch
2 times, most recently
from
0db80fa
to
de235b1
Compare
added 5 commits
June 13, 2017 16:10
This change adds ability to replicate dependency with index parameters iterated over arbitrary number of lists. For each dependency it is now possible to specify map of indexVariableName -> listExpression listExpression := range|item + [, listExpression] range := number '..' number item := STRING for example, if for "i: 1..3" the dependency will be replicated into 3 clones, each one of them having argument i set to value in range [1, 3] This also allows to consume N flow replicas by replicating the dependency that leads to the consumed flow
For sequential flow, each next replica is attached to the leafs of previous one so that they will be deployed sequentially
* stopChan is now passed to the graph finalizers so that deployment can be canceled on the final stages * never write to stopChan. The only correct way to cancel deployment is to close the channel * pass nil instead of real chanel for unit tests that do not cancel deployment
In some cases deployment could hang forever: * if graph vertex depends on vertex which will never be created (because of timeout or permanent error) * if graph vertex was set to be created only if parent fails, but it didn't Because deployment algorithm waits for all vertexes to be created if any one of them remained blocked, Deploy() is going to run forever blocking AC process from handling other deployment tasks. Also on-error processing could be triggered by intermediate resource status. For example it could happen, if resource status was obtained prior to resource.Create() call. Another case if resource was set to have several deployment attempts. If the first attempt fails on-error dependency becomes activated, but on the second attempt the deployment may succeed. This commit reworks error handling: * Resources which cannot be created or time out, marked with error. * Resources, that depend on failed resources also fail * Thus all graph vertexes eventually become unblocked and deployment finishes * on-error handling is done based on the final resource status/error * Deploy() now returns true if deployment succeeded. Deployment fails if any of resources (and their dependents) went into failed state except for cases, where they were skipped because all dependencies had on-error meta and parent resource didn't fail Also: * e2e tests were updated so that most of them wait for deployment to finish rather than just waiting for resource status. Thus now they also test that deployment doesn't hang * Graph vertex type (ScheduledResource) and it fields are not exported anymore. The same goes for some of dependency graph methods. * wait() method doesn't create unnecessary goroutines and channels
* Removes one second delay before scheduler starts to check if dependent resources can be created. This drastically speeds up unit tests and subsequent deployments when resources already exist and thus resource is created instantly * Caches dependencies and definitions during deployment. When one flow is consumed from another, they were all fetched from k8s for each replica of the inner flow + once for the outer flow. Now it happens only once. It also makes deployment more consistent in case if definitions were modified in k8s in the middle of deployment * Check dependency graph for cycles only once per deployment
istalker2
force-pushed
the
performance-improvements
branch
from
de235b1
to
bbf78fe
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resources can be created. This drastically speeds up unit tests
and subsequent deployments when resources already exist and thus
resource is created instantly
is consumed from another, they were all fetched from k8s for each
replica of the inner flow + once for the outer flow. Now it happens
only once. It also makes deployment more consistent in case if
definitions were modified in k8s in the middle of deployment
This change is