[For discussion] basic debug support for deployers #502
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently only STARTED is recognized and interpreted as module state `deployed`. This commit recognizes all the application instance states we are aware of, and interprets them as various module instance states. It also attempts to obtain application instance status irrespective of the reported state of the application, and ignores the exception that might be thrown.
Currently only a single module instance is deployed. This commit recognizes the count property on the module deployment request, and deploys that number of application instances.
* Initial version that generates reference documentation * Based on the configuration for `Spring Boot`
added spring-cloud-dataflow project url and organization attributes to the root pom.xml
If a deployment fails on CloudFoundry the resulting apps may be in the "DOWN" state. This state was not allowed for in the calculation of module status, which meant that the module was classed as 'unknown'. Modules in that status were not undeployed, hence the apps were orphaned on CF. This change adds the missing state, and hence allows such modules to be deleted as expected.
- When deploying stream, if the stream modules are partitioned, then set appropriate binding properties to handle the producer/consumer modules' partitioning properties. - Use the binding key `spring.cloud.stream.bindings.input/output.destination` for input/output binding keys so that partitioning is always supported with `Map` object as binding properties - For the consumer module: - Set `s.c.s.instanceCount`, `s.c.s.bindings.input.partitioned` with the assumption on `INSTANCE_INDEX`/`CF_INSTANCE_INDEX` would always be set for the module at run time for the target lattice/CF. - For the producer module: - Set `s.c.s.bindings.output.partitionCount` from the down stream module's count value (nextModuleCount) - Set `s.c.s.bindings.output.partitionKeyExpression`, `s.c.s.bindings.output.partitionKeyExtractorClass`, `s.c.s.bindings.output.partitionSelectorClass` and `s.c.s.bindings.output.partitionSelectorExpression` when required. - Infer upstream module's partition info
- adding set-env commands for use on PWS - adding env vars to use for deployment to PWS - Fixing command for generating docs
* Add banner for `Shell` * Add banner for `Admin` application
* Rearranging document structure * Adding module info and options * Adding Data Flow overview doc * Adding Data Flow getting started doc * Updating productname and copyright * Adding authors * Adding Spring Cloud Stream overview chapter * Updating to 1.0.0.M1 * Changing to use download commands for jars * Fixing missing admin download for YARN
* admin container tag is 1.0.0.M1 * reference module launcher container tag 1.0.0.M1
Added dependency for UriRegistry UriRegistry based on Redis Properties file for initial UriRegistry population
also populating the registry on startup by default (with "locations" and "enabled" as boot properties of the populator) The DelegatingResourceLoader includes: - MavenResourceLoader for 'maven:' - DefaultResourceLoader for 'file:' and 'http:'
- when a module is registered at runtime, it will be saved in ArtifactRegistry and registered with UriRegistry (if not null) - when a module is unregistered at runtime, it will likewise be removed from both - once all dataflow-server implementations use the new Deployer SPI, we will remove ArtifactRegistry altogether
* Add Restful endpoints to access Job features * Set SCDF to use SCT Snapshot * Add support for Task/Job association * Add support for JobExecution query * Add support for JobInstance query * Add support for JobStepExecution query * Restful results now match what was returned by XD but converted to SCDF format. * Add stepexecution to job list * Add TaskDefinitionRepository to TaskJobRepository to search for taskDefinitions * Remove spring-batch-admin from rest-client and resources * Update stepExecutionList to return proper page. * Remove mysql dependency * Remove XD based comments * Add Java Docs. Resolves spring-cloud#450
- AppRegistry wraps UriRegistry and returns AppRegistration - AppRegistration replaces ArtifactRegistration - AppRegistration defines a getResource() method - ModuleResolver is therefore no longer needed for resolving Resources - ModuleResolver can be removed with ModuleLauncher when it's no longer used
- New AppDeploymentRepository which is doing reverse mapping from AppDeploymentKey into a identifier returned from a deployer. - Only InMemoryAppDeploymentRepository for now. - AppDeploymentKey is a composition of StreamDefinition and ModuleDefinition which are used/needed when status() and undeploy() are used.# - Implemented hash/equal for StreamDefinition and ModuleDefinition for AppDeploymentKey to work as a key in maps. - StreamDefinitionController/StreamDeploymentController now uses this repository to backmap id from current in-flight StreamDefinition and ModuleDefinition in use. Review changes for AppDeploymentRepository - Add redis implementation. - AppDeploymentKey now used for either stream or task. - AppDeploymentKey now builds a unique key which is then used for backmapping in maps or hashes. - Removed middle KeyRepository as it's not needed anymore. - Added repository to TaskDeploymentController and update backmapping when task is launched. This mapping is not yet used for anything as more sophisticated task features are not yet implemented on a task infra. - RedisAppDeploymentRepositoryTests using RedisTestSupport from spring-cloud-stream-test-support-internal. Polishing - using String as deployment key
- Un-Ignore, polish and add use of RedisTestSupport. - Fixes spring-cloud#493
- now simply spring-cloud-dataflow-registry - moved the URI registry related classes there as well
- assuming status is undeployed when deployment id is null Fixes spring-cloud#499
|
Any updates on this PR status? |
|
@LyashenkoGS I am actually working on it and have a local prototype with possibly a simpler solution by adding fist class support for debugging to Local Deployer. Give me till next week as I am tied up now |
|
@LyashenkoGS One thing I forgot to mention which is already pushed. With local deployer you can now redirect logs to local server console which may not satisfy you fully with regard to debugging, but it will simplify the process quite a bit since you no longer have to go and tail multiple logs. |
|
Hi, Just wondering if anyone's planning to get this rolled out soon. Currently, I have been facing an issue on how to (remote) debug my apps deployed on SCDF and came across this PR. Appreciate someone taking care of this soon. Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Created two pull requests (one here on dataflow, the other on local deployer), these are for github issue #369 "Add debug capabilities to LocalAppDeployer".
With these two in place you can turn on debug by specifying "module.foo.debug-port=8800" as a deployment property.
The first pull request (on the deployer) recognizes a debug-port property and uses it to add a debug agent to the configuration for the Process it launches.
The second pull request (on dataflow) recognizes the deployment pattern module..debug-port=XXX and maps that to the debug-port property in the deployer.
This works but are the properties in the right place? Is it ok to have this as a deployment config property? (It is quite nice that turning debug on/off does not require meddling with the stream definition). What do we do for other deployers? Do we need a simpler 'debug=true/false' option alongside this for any that don't need a port, or do we need a deployer specific namespace subsection for this?
This resolves #369