Load templates dynamically from a database

[isaranchuk]

I really like jte syntax and its simplicity, but I'm wondering if jte is suitable for the following use cases:

1. Our system should be able to render templates that are stored in external datastore, e.g. relational DB.
2. Also we should be able to adjust templates without the need to restart or redeploy our application but only change template body in a DB

[edward3h]

You could make a custom implementation of the CodeResolver interface, which loads templates from the DB.

[isaranchuk]

@edward3h thanks, I've checked the source code more thoroughly and CodeResolver definitely should work.

[isaranchuk]

@edward3h I've implemented simple POC with custom CodeResolver to load templates from the DB and it works just fine when I'm running it locally in my IDE.

But I have troubles running my POC as a docker container inside K8s because generated classes couldn't be compiled, e.g.

/jte-classes/gg/jte/generated/ondemand/screens/JteGetPlansGenerated.java:2: error: package gg.jte.support does not exist                                                                                                                                                                │
│ import gg.jte.support.ForSupport;                                                                                                                                                                                                                                                       
│ /jte-classes/gg/jte/generated/ondemand/screens/JteGetPlansGenerated.java:6: error: cannot find symbol                                                                                                                                                                                   │
│     public static void render(gg.jte.TemplateOutput jteOutput, gg.jte.html.HtmlInterceptor jteHtmlInterceptor, java.util.List<com.sleepnumber.sdui.templates.model.Plan> plans) {                                                                                                       │
│   symbol:   class TemplateOutput                                                                                                                                                                                                                                                        │
│   location: package gg.jte 
....

Also it complains on my custom params (classes) inside templates.

And when I execute request one more time I can see

java.lang.ClassNotFoundException: gg.jte.generated.ondemand.screens.JteGetPlansGenerated

But I can see jte-classes/gg/jte/generated/ondemand/screens/JteGetPlansGenerated.java inside my docker container.

Some technical details about my setup:

1. Spring Boot
2. Java 11
3. Docker base image amazoncorretto:11-alpine-jdk

I'm wondering if jte supports on-demand templates when running inside docker container?

[edward3h]

It looks like the compiler doesn't have the correct classpath at the point it is trying to compile the template. It should be including the jte-runtime module. Unfortunately I don't know much about Spring Boot but I would guess the problem is related to how it loads classes. Or maybe you just need the extra dependency in your build.

It's unlikely to be related to Docker. It could be worth trying with eclipse-temurin:11-jdk instead just for a comparison.

[isaranchuk]

I definitely missed jte-runtime module in my pom.xml but after I added these jte dependencies

<dependency>
            <groupId>gg.jte</groupId>
            <artifactId>jte</artifactId>
            <version>2.1.2</version>
        </dependency>
        <dependency>
            <groupId>gg.jte</groupId>
            <artifactId>jte-runtime</artifactId>
            <version>2.1.2</version>
        </dependency>

I still can see the error I mentioned above.

It's a really weird behaviour and it's not clear how to fix it.

[casid]

jte-runtime comes as transitive dependency through jte, so it shouldn't make any difference.

How is your TemplateEngine initialized?

[isaranchuk]

@casid thanks for your help.
This is how I initialize template engine:

@Configuration
public class TemplateConfiguration {

  @Bean
  public TemplateEngine templateEngine(CodeResolver codeResolver) {
    return TemplateEngine.create(codeResolver, ContentType.Plain);
  }
}

Where I inject my custom CodeResolver to load templates from DB.

[casid]

Can you check what classloaders are used? There was a similar issue with Spring Boot before: #70

Just thought I share some concerns about your setup:

• You will rely on compiling Java code on your production system
• In case anybody gets access to the database, or is able to insert data to the templates table through another vulnerability, the system is instantly subject to a really, really bad RCE attack.

[isaranchuk]

@casid thanks, I understand concerns but still I'd like to have POC to meet different requirements.

These classloaders are used:

TomcatEmbeddedWebappClassLoader                                                                                                                                                                                                                                                         
│   context: ROOT                                                                                                                                                                                                                                                                         
│   delegate: true                                                                                                                                                                                                                                                                        │
│ ----------> Parent Classloader:                                                                                                                                                                                                                                                         │
│ org.springframework.boot.loader.LaunchedURLClassLoader@4c75cab9                                                                                                                                                                                                                         │
│ org.springframework.boot.loader.LaunchedURLClassLoader@4c75cab9

As a result of

System.out.println(Thread.currentThread().getContextClassLoader());
System.out.println(getClass().getClassLoader());

Where java application inside docker container is started as

ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"]

[isaranchuk]

@casid classloader was definitely a problem, so once I passed the correct loader it started to work.

BTW could you please share more details about RCE attack that is possible with jte in this case?

[casid]

Since jte uses plain Java for expressions, pretty much anything is possible.

For example, to end the application, whenever the malicious template is rendered:
!{System.exit(0);}

Or, encrypt all files on the filesystem or tables in the database, that the application has access to.

[isaranchuk]

@casid thanks for sharing more details on that.

My main requirement is to be able to load different versions of a template at runtime.
E.g. we have two different versions of User Profile screen layout: 1.0.0 or 1.1.0 and then based on the client version we can return proper screen layout.

Also templates will not be populated by the end-users but only by developers through the automated CI/CD pipeline.

With great power comes great responsibility, I mean jte feature-rich Java expressions, so maybe it worth to consider some simple template engines as well, e.g. Handlebars, just to reduce the attack surface.

[casid]

Are developers committing those templates to the version control of the project?

If so, you could only make the decision what template to use configurable, not the templates themselves. It's e.g. easy to check feature toggles in jte templates and do stuff differently.

In case you want to be able to do all this without an app deployment, you could also precompile all templates on your CI/CD and then upload the compiled templates and then replace the template engine (through TemplateEngine#reloadPrecompiled). This way you don't need a JDK on your production system.

And yes, if you plan to do it through the database, I'd consider using a dumber template engine where remote code execution is impossible (if that exists). I believe being able to execute arbitrary e.g. Handlebars templates on a system could be quite dangerous as well.

[isaranchuk]

@casid yes, the idea is store templates in git repo and introduce a new template version each time there's a change.
We need to be backward compatible where old client can still request older template version.

In case you want to be able to do all this without an app deployment, you could also precompile all templates on your CI/CD and then upload the compiled templates and then replace the template engine (through TemplateEngine#reloadPrecompiled). This way you don't need a JDK on your production system.

Interesting idea, I was thinking about that as well but then I'm wondering how I can resolve template version?
E.g. if in file system we have these templates:

/application/jte-classes/gg/jte/generated/precompiled/v1.0.0/screens/UserProfile.class
/application/jte-classes/gg/jte/generated/precompiled/v1.0.1/screens/UserProfile.class

Also I'm wondering if it's possible to load generated templates from external system, e.g. aws s3, etc?
Because if we're running our application in container then we should think about mounting some persistent volume or use some external datastore, e.g. s3, database, etc.

[casid]

I'm not sure what you're building, but I would just deploy the precompiled templates with the application and call it a day. Like, you probably don't store the Java files on an external system and load them dynamically, right?

And once and a while you're probably deleting old templates, since the whole data (Java classes) populating those templates aren't up to date anymore, too.

[casid]

I just noticed, that I didn't really answer your question.

how I can resolve template version?

You can use the version in your template name before you call render: render(version + "screens/UserProfile.jte", ...). Or, if those are separated class roots, you can have a TemplateEngine instance per version.

possible to load generated templates from external system, e.g. aws s3, etc?

You could download the precompiled class files from there to your server and then create a TemplateEngine instance from it.

As said before, this setup seems a bit wild and potentially dangerous (side loading executable code from DB or AWS). In that case an interpreted and slower template language that doesn't need to be compiled might be the better choice.

I still don't understand why you need to be able to produce results for old clients though. The beautiful thing about websites is that you deploy them and everything is up-to-date.

[isaranchuk]

@casid thank you for your answers, they're really helpful.