Document custom SCDF server setup

[ilayaperumalg]

We can add a section that describes how to set up a custom SCDF server using EnableDataFlowServer annotation and its configuration.

[sabbyanandan]

We have recently added a sample to demonstrate how to build a custom SCDF server with the Spring Boot app created from Spring Initializr.

The README can get some love in regards to granular steps and the explanation of specific steps to clarify what it includes already. It'd be nice also to update the module name to be more descriptive.

In other words, we do not want to create yet another sample, but instead, we could update what is available already.

[imod]

@ilayaperumalg @sabbyanandan many thanks for creating an issue on this topic! I'm trying to create a custom build based on the sample mentioned above, but I'm already stuck at the point where I don't know which tables should be created? It seems tables are not created by the application itself (as it is the case for spring-batch). Can you point my to where the ddl definitions are located? or how should this work?

[ilayaperumalg]

Hi @imod,

Thanks for checking this out.

The SCDF specific tables are created using Spring Data JPA as the SCDF schema entities are defined as JPA repositories. For instance, you can see the StreamDefinition defined here

[imod]

@ilayaperumalg @sabbyanandan I found another reason why I do need to create a custom scdf: when I follow the process as described on https://dataflow.spring.io/docs/installation/cloudfoundry/cf-cli/#installing-using-a-manifest, then the installation fails because flyway does not accept our MariaDB:

   2019-08-23T15:51:33.70+0200 [APP/PROC/WEB/0] OUT org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer' defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]: Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Unsupported Database: MariaDB 10.1
   2019-08-23T15:51:33.70+0200 [APP/PROC/WEB/0] OUT 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1778)
   2019-08-23T15:51:33.70+0200 [APP/PROC/WEB/0] OUT 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
   2019-08-23T15:51:33.70+0200 [APP/PROC/WEB/0] OUT 	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
   2019-08-23T15:51:33.70+0200 [APP/PROC/WEB/0] OUT 	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
   2019-08-23T15:51:33.70+0200 [APP/PROC/WEB/0] OUT 	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)

Locally I do have it running with the exact same DB version (10.1.38-MariaDB), but I had to create all the tables manually by telling hibernate to create the tables (spring.jpa.hibernate.ddl.auto=update), because the DDL is not documented (I created an issue for this). In the same direction I also created (spring-cloud/spring-cloud-dataflow#3461).

Another point I'm confused about is skipper, here it says skipper is optional, but here it is not noted as optional. If it is optional, what functionality will I lose? Only the blue/green rollup deployments or will SCDF not be able to deploy batch/task jobs at all anymore?

[sabbyanandan]

Hello, @imod. Thanks for the details! We will review those issues to see what can be done there.

As for Skipper being optional or not, starting from SCDF 2.0 and onwards, Skipper is a required component. Unfortunately, the docs that you're referring to (though it points to current) is v1.7.4 - the last point release on the 1.x.x release line. If you're trying to build and extend SCDF, please use 2.2.0 GA, which is the latest production release (2.2.1 shipping soon, too). We will also have to fix the current version of the URL to point to the newest GA docs.

Just curious, how did you discover the docs URL? We will have to fix all of the references as well.

[sabbyanandan]

Update: It appears the current docs URL is rightly redirecting to the latest 2.2.0 GA release. I didn't pay attention to the URL that you shared, though. You're explicitly going to spring-cloud-dataflow-server-cloudfoundry, and that's why it is pointing to 1.7.4.

One more time, starting from 2.0 and onwards, Skipper is a required component. And likewise, we have merged all the supported SCDF's platform-specific autoconfiguration's into a single-server, so you can now use spring-cloud-dataflow-server in local, CF and K8s. The same binary that is.

[imod]

thanks @sabbyanandan that's very valuable information! I hope you don't mind me having another question: we already run an instance of spring boot admin and I think it would be great to combine the SBA mit SCDF in one app, do you know if this is feasible? e.g. I can define path prefix for SBA (the same is also possible for cloud config server) - if this would be possible for SCDF, that would be great.

[imod]

@sabbyanandan section 9.1 - feature toggles indicates Skipper is only required for Streams, as I'm only interested in running Batch jobs, I might not need Skipper - is this true?
The whole reason why I look at SCDF is because spring batch admin is discontinued and we need some UI for batch also in the future.

[sabbyanandan]

My apologies for the delay. Correct, if you don't want to use streaming features in SCDF, you can disable it using the respective feature-toggle. When you do that, you do not need Skipper indeed.

As for SBA vs. SCDF, the aim was to move all of the SBA features in SCDF, and that is feature complete. And, only after that, SBA was pulled off of maintenance. That said, SCDF is a product and it has opinions in regards to how it bootstraps with the streaming, batch, and analytics features - we have been working on this for >5yrs now. So, any standalone SBA footprint that you have from the past, it won't automatically fold into SCDF as-is. Your net-new Task/Batch apps can be configured and launched from SCDF. If you have an existing batch-job workload, you must convert it to a Spring Cloud Task, and spawn it as Task definitions in SCDF, as well.

[imod]

@sabbyanandan thanks, I fully understand that batch apps now have to be spawned via Task definitions and that's all fine. I would love to follow the new architecture and programming model. The thing is, that we face a couple of problems which I'm currently not able to solve. e.g.

• on our cloudfoundry infrastructure, there is no OAuth server available and we would like to simply use basic auth
• we are not able to run task with java 11 not able to run task with java 11 / allow definition of environment variables for a task when launched on cloudfoundry spring-cloud/spring-cloud-dataflow#3467
• SCDF does not handle buildpacks correctly A Task-specific buildpack override is ignored on CF spring-cloud/spring-cloud-dataflow#3466

I'm aware that specially my first point might be controversial and maybe even naiv to expect it to work. I'm have a custom build of SCDF with basic auth and I can login and look around, but for some reason it fails to let my register new apps and I have no idea whats causing the issue.

If I do manage to get a custom build working, then I would love to contribute the learnings back in form of documentation and resolve #119 myself :) But I'm afraid I probably need some more help to get everything working.

[sabbyanandan]

We have added a sample for custom SCDF builds, and we have been keeping it up-to-date at the time of each new minor/major release. Unfortunately, we didn't update the issue with this info sooner, though.

Anyway, if we can help with anything else on top of it, please let us know.

[sabbyanandan]

Here's the sample -> https://github.com/spring-cloud/spring-cloud-dataflow-samples/tree/master/custom-dataflow-builds