Using JSON instead of Properties

[marcellodesales]

Hi there,

I have a requirement to share configuration between backend and frontend layers of applications using JSON format. That way, the front-end can load and use JSON directly.

I need the backend services to support loading properties from JSON as it is done with YAML http://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html#boot-features-external-config-yaml.

Is there a quick way to support storing configuration using JSON instead of YAML and properties files?

thanks
Marcello

[philwebb]

We prefer to keep the issue tracker for bugs only, this kind of question would be better suited to stackoverflow.com.

Take a look at the PropertySourceLoader interface, you can probably implement this and add it to spring.factories to support new file types.

[shakuzen]

@philwebb I believe the intention was for this to be a feature request / enhancement request for Spring Boot to natively support loading PropertySources from JSON. See spring-cloud/spring-cloud-config#233 for background.

[marcellodesales]

Thank you @philwebb @shakuzen

@shakuzen is correct... This is more of a feature request than a question... I will see what I can learn with the PropertySourceLoader...

[philwebb]

I'm afraid I'm somewhat reluctant to make json configuration a first class citizen with Spring Boot but I'm happy to tweak anything so that you can support it in your own application.

[marcellodesales]

@philwebb Shouldn't we be able to declare dependencies we want. Ok, so, I'm part of the Services Platform at Intuit and we have to support both Java and Node.js applications. We have our own GitHub Enterprise and we are supporting around 20 Nodejs micro-services from different groups... Why can't you provide a hook that I can just add my own configuration management?

I have a simple implementation that does not depend on loading multiple profiles. We can provide support for multiple profiles inside the same json object if we define that the first elements of the object are actual profiles.

I see a lot of value in supporting yet another file format, as other services in different programming languages could take advantage of. Would it be possible to get a common ground in supporting a custom/external dependency? Is there a way to inject that extra spring.factories?

[philwebb]

@philwebb Shouldn't we be able to declare dependencies we want. Ok, so, I'm part of the Services Platform at Intuit and we have to support both Java and Node.js applications. We have our own GitHub Enterprise and we are supporting around 20 Nodejs micro-services from different groups... Why can't you provide a hook that I can just add my own configuration management?.

I believe that the hooks are already there. You should be able to create your own PropertySourceLoader implementation, package it in a jar with a spring.factories and have your own apps support json (see below).

I have a simple implementation that does not depend on loading multiple profiles. We can provide support for multiple profiles inside the same json object if we define that the first elements of the object are actual profiles.

Have you managed to integrate this with Spring Boot already? Would you be able to share the code?

I see a lot of value in supporting yet another file format, as other services in different programming languages could take advantage of.

I'm not totally discounting it out of hand, and obviously you have a genuine need for this. The problem is, if we add this to the code we need to consider that Spring Boot itself becomes more complex. We need to scan more files when the application starts; we need to consider profile support and we need to update all the documentation. On top of that there is IDE support in STS which ideally would also need to be updated. It's a lot of additional overhead for a configuration format that I don't think is all that common.

Would it be possible to get a common ground in supporting a custom/external dependency? Is there a way to inject that extra spring.factories?

Yes, absolutely! This would be my preferred approach. We provide support for Properties and YAML out of the box but there are extensions that allow you to plug in whatever you need. I believe this should work already. What you need to do is:

• Create a library jar (e.g. spring-boot-intuit.jar)
• Add a class that implements org.springframework.boot.env.PropertySourceLoader

public class JsonPropertySourceLoader implements PropertySourceLoader {

    public String[] getFileExtensions() {
        return new String[] {"json"};
    }

    public PropertySource<?> load(String name, Resource resource, String profile) throws IOException {
        // load json, create and return a PropertySource
    }

}

• Add src/main/resources/META-INF/spring.factories

# PropertySource Loaders
org.springframework.boot.env.PropertySourceLoader=\
com.intuit....JsonPropertySourceLoader

You should then just be able to add spring-boot-intuit.jar to any project to get json configuration support.

[philwebb]

I'll re-open this and mark it as waiting-for-votes. Let's see how much demand there is for json. If you have trouble implementing the approach described above please let me know.

[snicoll]

Please note that IDEs currently support properties and yaml format (to some extend). If we were to have an official support for json, IDEs should also catch up on that format.

[marcellodesales]

@snicoll Thanks for the note... Yeah, I'm looking for an alternative that provides support for the deployed services at the moment. I would think we would work with the community to get the support for IDEs and hopefully an official support of the libraries.

@philwebb I greatly appreciate your reply... I've been super busy this week the last 40hrs :S and I hadn't got back to this... I had implemented exactly what you described on Sunday but did not get time to reply here... So far, here's what I got:

pom.xml Dependency

For loading JSON.

            <dependency>
                   <groupId>com.fasterxml.jackson.core</groupId>
                   <artifactId>jackson-core</artifactId>
                   <optional>true</optional>
             </dependency>

JsonPropertiesLoader

There are a lot of restrictions in this version:

• Json documents do not have the notion of multiple documents.
  • We can have a convention over the environments/profiles, but I think that's not the case.
• I copied code from the YAMLProcessor :) But it follows the same structure for JSON. If the copied method were a utility, I would not need to do so.

/*
 * Copyright 2012-2015 the original author or authors.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.springframework.boot.env;

import java.io.IOException;
import java.util.Collection;
import java.util.Collections;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Map.Entry;

import org.springframework.beans.factory.config.YamlProcessor;
import org.springframework.beans.factory.config.YamlPropertiesFactoryBean;
import org.springframework.core.env.MapPropertySource;
import org.springframework.core.env.PropertySource;
import org.springframework.core.io.Resource;
import org.springframework.util.ClassUtils;
import org.springframework.util.StringUtils;

import com.fasterxml.jackson.databind.ObjectMapper;

/**
 * Strategy to load '.json' files into a {@link PropertySource}.
 *
 * @author Marcello de Sales
 */
public class JsonPropertySourceLoader implements PropertySourceLoader {

    @Override
    public String[] getFileExtensions() {
        return new String[] { "json" };
    }

    @Override
    public PropertySource<?> load(String name, Resource resource, String profile)
            throws IOException {
        if (ClassUtils.isPresent("com.fasterxml.jackson.databind.ObjectMapper", null)) {
            Processor processor = new Processor(resource, profile);
            Map<String, Object> source = processor.process();
            if (!source.isEmpty()) {
                return new MapPropertySource(name, source);
            }
        }
        return null;
    }

    /**
     * {@link YamlProcessor} to create a {@link Map} containing the property values.
     * Similar to {@link YamlPropertiesFactoryBean} but retains the order of entries.
     */
    private static class Processor {

        private Resource resource;

        Processor(Resource resource, String profile) {
            this.resource = resource;
        }

        @SuppressWarnings("unchecked")
        public Map<String, Object> process() {
            final Map<String, Object> result = new LinkedHashMap<String, Object>();
            Map<String, Object> map;
            try {
                map = new ObjectMapper().readValue(this.resource.getFile(), LinkedHashMap.class);
                result.putAll(getFlattenedMap(map));

            } catch (Exception e) {
                e.printStackTrace();
            }
            return result;
        }

        /**
         * Copied from
         * https://github.com/spring-projects/spring-framework/blob/master/spring-beans/
         * src/ main/java/org/springframework/beans/factory/config/YamlProcessor.java
         * @param source
         * @return Flattened map of the key.subkey=value
         */
        private final Map<String, Object> getFlattenedMap(Map<String, Object> source) {
            Map<String, Object> result = new LinkedHashMap<String, Object>();
            buildFlattenedMap(result, source, null);
            return result;
        }

        private void buildFlattenedMap(Map<String, Object> result,
                Map<String, Object> source, String path) {
            for (Entry<String, Object> entry : source.entrySet()) {
                String key = entry.getKey();
                if (StringUtils.hasText(path)) {
                    if (key.startsWith("[")) {
                        key = path + key;
                    } else {
                        key = path + "." + key;
                    }
                }
                Object value = entry.getValue();
                if (value instanceof String) {
                    result.put(key, value);

                } else if (value instanceof Map) {
                    // Need a compound key
                    @SuppressWarnings("unchecked")
                    Map<String, Object> map = (Map<String, Object>) value;
                    buildFlattenedMap(result, map, key);

                } else if (value instanceof Collection) {
                    // Need a compound key
                    @SuppressWarnings("unchecked")
                    Collection<Object> collection = (Collection<Object>) value;
                    int count = 0;
                    for (Object object : collection) {
                        buildFlattenedMap(result,
                                Collections.singletonMap("[" + (count++) + "]", object),
                                key);
                    }

                } else {
                    result.put(key, value == null ? "" : value);
                }
            }
        }
    }
}

Thanks a lot for clarifying and providing the steps above... I will try to follow them by Friday and then I can provide more feedback...

As Intuit is hosting a Spring Cloud Config connected with out GitHub Enterprise, this will bring a lot of value to the Node.js internal Community. Thanks a lot for listening!

[philwebb]

One thing that @dsyer noted is that YAML is a superset of JSON so you might get away with extending YamlPropertySourceLoader and overriding the getFileExtensions() method.

[marcellodesales]

Hey @philwebb, According to SnakeYAML owner, https://bitbucket.org/asomov/snakeyaml/issues/316/json-configuartion#comment-22046776, SnakeYAML implements YAML spec 1.1 and, therefore, that's NOT a complete superset of JSON, which is YAML 1.2. That's reason why I implemented this way... As he (snakeYaml's owner) suggested, it would take someone's energy to implement the YAML 1.2 spec :( I don't have time at the moment to play on it :(

[wilkinsona]

Closing for the same reasons as described here.

[bluejaguar]

FYI - for those coming here in 2017 or beyond, from what I can tell this solution JsonPropertiesLoader does not work (with Dalston.SR3). Initialization through PropertySourceBootstrapConfiguration ignores .json files and this custom JsonPropertiesLoader is never called, even with references in spring.factories for org.springframework.boot.env.PropertySourceLoader.
It seems that there are many, many use cases for custom config sources that want to utilize the plumbing in SCC but the current paradigm is locked for .properties and .yml files ?

[marcellodesales]

@bluejaguar @philwebb @wilkinsona I got a workaround as follows:

• Create a yml file and add json content
• The file will be loaded based on the specs and what @philwebb mentioned

For instance, I have myapp.yml and I add { "name" : "My APp" } and it works to load.