diamond-miner

a java config system base on taobao diamond, main changes included:

• API is simplified to be more friendly.
• All files charset changed from GBK to UTF-8.
• Logging system changed from commons logging to slf4j.
• All code were cleaned and refactored.
• Description and valid fields were added for convenience.
• Diamond server disk files are rooted as use home directory.
• A cache was added to client based on diamond and guava cache.
• Some other changes.

Usage

• Setup mysql database and tables
• Startup diamond-server
• Try to use diamond-client

Setup mysql database and tables

• create database diamond

    create database diamond;

• create table

-- for MySQL
create table diamond_stones (
    id varchar(255) not null,
    data_id varchar(255) not null,
    group_id varchar(128) not null,
    content longtext not null,
    description longtext default null,
    valid tinyint(1) not null default 1,
    gmt_create datetime not null,
    gmt_modified datetime not null,
    primary key (id),
    unique key uk_diamond_datagroup (data_id,group_id)
) default charset=utf8;

-- for ORACLE 
create table diamond_stones (
    id varchar2(400) not null primary key,
    data_id varchar2(255) not null,
    group_id varchar2(128) not null,
    content long not null,
    description varchar2(1024),
    valid number(1,0) default 1,
    gmt_create date not null,
    gmt_modified date not null
);

create index idx_diamond_stones on diamond_stones(data_id, group_id);

• create user

-- for MySQL
create user 'diamond'@'%' identified by 'diamond';
grant all privileges on diamond.* to diamond@'%';

create user 'diamond'@'localhost' identified by 'diamond';
grant all privileges on diamond.* to diamond@'localhost';

Setup diamond-server

1. download the diamond-server code from github
2. run the command:

mvn package && cd target && java -jar diamond-server-0.0.8-exec.war -httpPort 17002

• Cluster example
  1. java -jar diamond-server-0.0.8-exec.war -httpPort 18001
  2. java -jar diamond-server-0.0.8-exec.war -httpPort 18002
  3. Add config: dataId=nameservers, group=admin, content=localhost:18001 localhost:18002

The default mysql connection is:

db.driver=com.mysql.jdbc.Driver
db.url=jdbc:mysql://localhost:3306/diamond?useUnicode=true&&characterEncoding=UTF-8&connectTimeout=1000&autoReconnect=true
db.user=diamond
db.password=diamond
db.initialSize=10
db.maxActive=10
db.maxIdle=5
db.maxWait=5
db.poolPreparedStatements=true

If you have the different mysql ip, user or passoword, you can have several ways to change your connection info:

• put a diamond-jdbc.properties in your current directory(same directory with diamond-server-0.0.X.war).
• update WEB-INF/classes/diamond-jdbc.properties in diamond-server-0.0.X.war.
• put a diamond-jdbc.properties in ~/.diamond-server/。

After you setup diamond-server successfully, you can open http://localhost:17002/diamond-server to login in diamond-server console. The default username and password is admin/admin.

Try to use diamond-client

• Setup connection info for client
  • Create diamond-client.properties in the classpath root or in ~/.diamond-client/.
  • Add one line to diamond-client.properties

DiamondServer.address=localhost:17002

or cluster way

DiamondServer.address=localhost:18001 localhost:18002

• Simple use examples

String foo = new Miner().getString("foo"); // foofoo
String bar = new Miner().getStone("my_group", "bar"); // barbar

int defaultTimeout = 10;
int timeout = new Miner().getInt("timeout", defaultTimeout);

• Add a listener to repsonse config changing.

DiamondManager diamondManager = new DiamondManager("foo");
diamondManager.addDiamondListener(new DiamondListener() {
    @Override
    public Executor getExecutor() {
        return null;
    }

    @Override
    public void accept(DiamondStone diamondStone) {
        System.out.println("DiamondListener:" + diamondStone.getContent());
    }
});

String diamond = diamondManager.getDiamond();

• Use substitution (Supported by DiamondMiner, and N/A in DiamondManager)

String foobar = DiamondMiner.getString("foobar");
// ${foo}-${bar} will substituted to foofoo-barbar

• Use diamond cache

  1. Create a cache updater class, and it should implement Callable interface.

      package org.n3r.diamond.client;
     
      import org.n3r.diamond.client.cache.ParamsAppliable;
      import java.util.Arrays;
      import java.util.Date;
      import java.util.concurrent.Callable;
     
      public class DemoUpdater implements Callable<String>, ParamsAppliable {
          private String param;
     
          @Override
          public String call() {
              return param + new Date();
          }
     
          @Override
          public void applyParams(String[] params) {
              this.param = Arrays.toString(params);
          }
      }
     

  2. Add a config like:

      dataId=foo, group=DEFAULT_GROUP,
      content=@org.n3r.diamond.client.DemoUpdater("Hello world") @TimestampMark("2013-10-09 10:37:08.123")
     

  3. Get the cache:

      String cache = DiamondMiner.getCache("foo");
     

  4. Update the cache manually, any change of config will cause cache updated.

      content=@org.n3r.diamond.client.DemoUpdater("Hello world") @TimestampMark("2013-10-09 10:37:10.234")
      or
      content=@org.n3r.diamond.client.DemoUpdater("你好，中国") @TimestampMark("2013-10-09 10:37:10.234")
     

Notes

• garbled characters on TOMCAT console: add -Dfile.encoding=UTF-8 to VM options
• check charset in MYSQL :

show create database diamond;
show create table diamond_stones;

set names utf8;
ALTER DATABASE `diamond` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
ALTER TABLE `config_info` DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;

Rantionale

• The client and server local store tree:

    ~/.diamond-client              ~/.diamond-server
    |____config-data               |____config-dump
    |____DiamondServer.address     | |____admin
    |____snapshot                  | | |____users.diamond
    | |____DEFAULT_GROUP           | |____DEFAULT_GROUP
    | | |____foo.cache             | | |____bar.diamond
    | | |____foo.diamond           | | |____foo.diamond
    | | |____SOLR_URL.diamond      | | |____SOLR_URL.diamond

• The client api will lookup config in privileged by order:
  • The local disaster recovery dir(~/.diamond-client/config-data)
  • Httpclient to diamond-server if local cache missed
  • Local snapshot

So if the diamond-server died, the client can still work by snapshot. And even when the snapshot is not available, the local disaster recovery dir can be used temporarily.

• The client has a standalone thread to update configs to diamond-server by comparing those md5 periodically per 15 seconds.
• The server will dump all the config from database periodically per 10 minutes.
• More links:
  • diamond专题（一）- 简介和快速使用
  • diamond专题（二）- 核心原理介绍
  • diamond专题（三）- diamond架构
  • diamond专题（四）- 容灾机制
  • ZooKeeper和Diamond有什么不同