Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[proposal] Support for 'standalone' and 'application group' support #915

Closed
donovanmuller opened this issue Oct 2, 2016 · 4 comments
Closed
Assignees

Comments

@donovanmuller
Copy link
Contributor

donovanmuller commented Oct 2, 2016

Currently there are four different types of applications:

  • source
  • processor
  • sink
  • task

This issue is to track the proposal of adding an additional application type, standalone.
A standalone application defines how a potentially non Spring Cloud Stream/Task Spring Boot app (no binder needed) can be deployed in a Data Flow context.

The standalone application type could be useful for:

  • Apps that supplement Streams/Tasks and could be deployed using the same tooling. For instance deploying a custom Spring Cloud Config Server, etc.
  • Avoids having to implement and potentially deploy a separate "deployer server" for deploying standalone apps using the Spring Cloud Deployer abstraction
  • Completes what it says on the box, "cloud-native orchestration service for composable microservice applications on modern runtimes" 😁

I have taken a stab at a basic implementation on my fork (standalone branch): donovanmuller@64da383. In addition, I've written a blog post demonstrating how the new standalone application type could work.

As mentioned in the post, we are going to use this internally to faciliate a consistent way of deploying both streams, tasks and current Spring Boot apps from the same tooling.

However, I'm not sure if this proposed new application type fits into the grander scheme of things for Data Flow? This issue is to open a conversation around whether or not it has merit.
Given the above, I've not opened a PR for my basic implementation, although would be more than happy to contribute as much as possible.

@donovanmuller donovanmuller changed the title [proposal] Introduce a new 'standalone' application type for deploying non Stream absed apps [proposal] Introduce a new 'standalone' application type for deploying non Stream based apps Oct 2, 2016
@sabbyanandan
Copy link
Contributor

Hi, @donovanmuller - Thanks for this proposal. 😄

There's a spike in-progress in the spring-cloud-deployer project [#99] with the scope to deploy any arbitrary application and as well, the support for application-grouping. This deployment option would eventually bubble up to SCDF's DSL/GUI, too.

@markfisher has been experimenting with the design and it'd be great to collaborate with you on this feature.

@geoand
Copy link

geoand commented Nov 23, 2016

👍 for this feature!

@donovanmuller donovanmuller changed the title [proposal] Introduce a new 'standalone' application type for deploying non Stream based apps [proposal] Support for 'standalone' and 'application group' support Nov 25, 2016
@donovanmuller
Copy link
Contributor Author

To add to the standalone application type, I have also drafted a take on the Application Group feature tracked by spring-cloud/spring-cloud-deployer#99 (albeit with slightly different requirements). This was added because we needed the ability to push a single artifact through our CI/CD pipeline (facilitated by the spring-cloud-dataflow-maven-plugin) that would enable the deployment of multiple streams/standalone apps/ etc. related to a logical solution.

There is a detailed write up about it in this blog post.

Having added these features and with the need for possibly more extension in the future, I'd like to get feedback on working on a plugin architecture that would allow extensions like this to be externally bolted on to the core data flow project, thereby not relying on the core project to adopt new features that might not make sense for the roadmap of the project and additionally, community forks need not be constantly rebased with upstream changes.

The plugin architecture could possibly provide lifecycle hooks.
So hooks into things like parsing the (with possible extensions) DSL, additional application types, custom definition types, registration, deployment and so forth.

With this capability added to the core data flow project, composing a new data flow server project with additional custom functionality, would simply require a dependency on data flow core and additional plugins. The configuration could be achieved through the usual Spring (auto) configuration mechanisms.

@sabbyanandan Would this be something that the team would be interested in supporting, if contributed?

@sabbyanandan
Copy link
Contributor

The finalized design proposal is here for review. We will be addressing the feature capabilities in incremental chunks and the first step in the process will be tackled via #1122 and #1123. For any other open items that need to be discussed, let's continue the conversation either in the open issues or google doc. I will close this issue for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants