Skip to content

Develop #191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
robinhood-jim merged 2 commits into from
Dec 10, 2024
Merged

Develop #191

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8e8b64c
modify and add customer FileSystemAccessor
robinhood-jim Dec 9, 2024
b7d6711
optimize code
robinhood-jim Dec 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 73 additions & 37 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,40 +8,85 @@ Slightly Framework design to support Spring based java or Bigdata program.

1.Introduction

I.This project is base on Spring Framework and has four modules:
|----------------------------------------------------------------------------------------------------|
| Module | Description |
|----------------------------------------------------------------------------------------------------|
| Core | the core class include data access layer basic class(model,dao,service) and etc. |
|----------------------------------------------------------------------------------------------------|
| Comm | FileSystem Access tool(local/vfs),support FileFormat(csv/xml/json/avro/parquet/protobuf)|
| | ,support Compress Format(gzip/bzip2/snappy/lzo/zip/lzma/lz4) |
| | ,read and write excel,read word or PowerPoint |
|----------------------------------------------------------------------------------------------------|
|Hadooptool|FileSystem Access tool(hdfs), comm tool to access to HDFS,Hbase,Hive,Mongdb and etc |
|----------------------------------------------------------------------------------------------------|
|Example |springmvc config based and spring boot based Example; |
|----------------------------------------------------------------------------------------------------|
|Web |struts1,struts2 and springmvc support web component and required class. |
|----------------------------------------------------------------------------------------------------|
|Webui |Spring Boot with Oauth2 Thymeleaf Example; |
|----------------------------------------------------------------------------------------------------|
|Estool | ElasticSearch Comm Query tool |
|----------------------------------------------------------------------------------------------------|
|Tracer | Zipkin Brave tracing，Can trace All Database and Record parameters |
|----------------------------------------------------------------------------------------------------|

II. Special feature
I.This project is base on Spring Framework and has below modules:
|----------------------------------------------------------------------------------------------------------------|
| Module | Description |
|----------------------------------------------------------------------------------------------------------------|
| Core | the core class include data access layer basic class(model,dao,service) and etc. |
|----------------------------------------------------------------------------------------------------------------|
| Comm | FileSystem Access tool(local/vfs),support FileFormat(csv/xml/json/avro/parquet/protobuf)|
| | ,support Compress Format(gzip/bzip2/snappy/lzo/zip/lzma/lz4) |
| | ,read and write excel,read word or PowerPoint |
|----------------------------------------------------------------------------------------------------------------|
|Hadooptool |FileSystem Access tool(hdfs), comm tool to access to HDFS,Hbase,Hive,Mongdb and etc |
|----------------------------------------------------------------------------------------------------------------|
|Example |springmvc config based and spring boot based Simple framework Example; |
|----------------------------------------------------------------------------------------------------------------|
|Web |struts1,struts2 and springmvc support web component and required class. |
|----------------------------------------------------------------------------------------------------------------|
|Webui |Spring Boot with Oauth2 Thymeleaf Example; |
|----------------------------------------------------------------------------------------------------------------|
|Estool | ElasticSearch Comm Query tool |
|----------------------------------------------------------------------------------------------------------------|
|Tracer | Zipkin Brave tracing，Can trace All Database and Record parameters |
|----------------------------------------------------------------------------------------------------------------|


It is available under the terms of either the Apache Software License 2.0 or the Eclipse Public License 1.0.

2.Support Features
I. Construct Simple Java FrameWork
contain use sysrole and relation with customer privilege to use and roles;
1.xml based standard frame : see example/config-example
2.spring based standard frame: see example/boot-example

II. Bigdata supprot
hadooptool:
HDFS tool: com.robin.hadoop.hdfs can access HDFS with kerberos security
Hbase tool: com.robin.hadoop.hbase hbase tool
Cassandra tool : CassandraUtils

III. BigData common file format read/write tools support(including compress type support)
AVRO
PARQUET
ORC
PROTOBUF
CSV
XML
JSON
ARFF(weka format)

IV. File storage and Cloud storage support
LocalFileSystem
ApacheVFS
HDFS
Amazon S3
Aliyun OSS
Tencent COS
Apache Kafka
RabbitMq

V. Iterable and wirtable support intergate Storage and file format
mix storage and File format to support cross storage read/write

VI. Spring cloud support
WebUI simple webui base on dhtmlxGrid 5.1 with spring boot native
related project in my another project microservices

VII. Zipkin Intergation
trace sub project aimed to support All database to be tracable and can record query parameters.

VIII. Special feature
a.A user defined xml Query config system,similar to mybatis,but easy config.
b.Support defined annotation or jpa annotation in JdbcDao with ORM.
c. BaseAnnotationService can access DB with minimize code,and use transaction with annotation.
d.A common db access meta and util,can access all kind of db.
e.Spring cloud based WebUI
f.support Hadoop plateform

It is available under the terms of either the Apache Software License 2.0 or the Eclipse Public License 1.0.
2.Development
g. Excel read write utils support auto merge columns and customer header define.


3.Development

I.Model Layer:Simple ORM tool, Support JAVA JPA or my BaseObject Annotation
Demostration：(support Composite primary key)
Expand Down Expand Up @@ -221,14 +266,5 @@ Slightly Framework design to support Spring based java or Bigdata program.

upon feature aim to simplify the work to develop standard MVC java code.

II. Bigdata supprot
hadooptool:
HDFS tool: com.robin.hadoop.hdfs can access HDFS with kerberos security
Hbase tool: com.robin.hadoop.hbase hbase tool
Cassandra tool : CassandraUtils

III. Spring cloud support
WebUI simple webui base on dhtmlxGrid 5.1 with spring boot native
related project in my another project microservices


3 changes: 1 addition & 2 deletions common/src/main/java/com/robin/comm/util/xls/ExcelProcessor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,8 +232,7 @@ public static Object readValue(Cell cell, String type, DateTimeFormatter format,
break;
case Const.META_TYPE_BIGINT:
if (!StringUtils.isEmpty(str)) {
Double d = Double.valueOf(str);
strCell = d.longValue();
strCell = Double.valueOf(str).longValue();
} else {
strCell = 0L;
}
Expand Down
4 changes: 4 additions & 0 deletions common/src/main/java/com/robin/core/fileaccess/iterator/AbstractFileIterator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -144,4 +144,8 @@ public String getIdentifier() {
public AbstractFileSystemAccessor getFileSystemAccessor(){
return accessUtil;
}

public void setAccessUtil(AbstractFileSystemAccessor accessUtil) {
this.accessUtil = accessUtil;
}
}
6 changes: 6 additions & 0 deletions common/src/main/java/com/robin/core/fileaccess/iterator/AbstractResIterator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package com.robin.core.fileaccess.iterator;

import com.robin.core.base.exception.OperationNotSupportException;
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
import com.robin.core.fileaccess.meta.DataCollectionMeta;
import com.robin.core.fileaccess.meta.DataSetColumnMeta;
import org.slf4j.Logger;
Expand Down Expand Up @@ -51,4 +52,9 @@ public void setReader(BufferedReader reader) {
public void setInputStream(InputStream inputStream) {
throw new OperationNotSupportException("");
}

@Override
public void setAccessUtil(AbstractFileSystemAccessor accessUtil) {
throw new OperationNotSupportException("");
}
}
1 change: 1 addition & 0 deletions common/src/main/java/com/robin/core/fileaccess/iterator/ArffFileIterator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

import com.google.common.collect.Lists;
import com.robin.core.base.util.Const;
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
import com.robin.core.fileaccess.meta.DataCollectionMeta;
import com.robin.core.fileaccess.meta.DataSetColumnMeta;
import org.springframework.util.CollectionUtils;
Expand Down
3 changes: 3 additions & 0 deletions common/src/main/java/com/robin/core/fileaccess/iterator/IResourceIterator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
package com.robin.core.fileaccess.iterator;
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;

import java.io.BufferedReader;
import java.io.Closeable;
import java.io.InputStream;
Expand All @@ -11,4 +13,5 @@ public interface IResourceIterator extends Iterator<Map<String,Object>>, Closeab
String getIdentifier();
void setInputStream(InputStream inputStream);
void setReader(BufferedReader reader);
void setAccessUtil(AbstractFileSystemAccessor accessUtil);
}
1 change: 1 addition & 0 deletions common/src/main/java/com/robin/core/fileaccess/iterator/JsonFileIterator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import com.google.gson.stream.JsonReader;
import com.robin.core.base.util.Const;
import com.robin.core.convert.util.ConvertUtil;
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
import com.robin.core.fileaccess.meta.DataCollectionMeta;
import com.robin.core.fileaccess.meta.DataSetColumnMeta;
import org.springframework.util.CollectionUtils;
Expand Down
1 change: 1 addition & 0 deletions common/src/main/java/com/robin/core/fileaccess/iterator/PlainTextFileIterator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import com.robin.core.base.util.Const;
import com.robin.core.base.util.StringUtils;
import com.robin.core.convert.util.ConvertUtil;
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
import com.robin.core.fileaccess.meta.DataCollectionMeta;
import com.robin.core.fileaccess.meta.DataSetColumnMeta;
import org.springframework.util.ObjectUtils;
Expand Down
3 changes: 2 additions & 1 deletion common/src/main/java/com/robin/core/fileaccess/iterator/TextFileIteratorFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,8 @@ public static AbstractFileIterator getProcessReaderIterator(DataCollectionMeta c
Class<? extends IResourceIterator> iterclass=fileIterMap.get(fileType);
try {
if (!ObjectUtils.isEmpty(iterclass)) {
iterator = (AbstractFileIterator) iterclass.getConstructor(DataCollectionMeta.class, AbstractFileSystemAccessor.class).newInstance(colmeta,utils);
iterator = (AbstractFileIterator) iterclass.getConstructor(DataCollectionMeta.class).newInstance(colmeta);
iterator.setAccessUtil(utils);
}
iterator.beforeProcess();
}catch (Exception ex){
Expand Down
6 changes: 5 additions & 1 deletion common/src/main/java/com/robin/core/fileaccess/iterator/XmlFileIterator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@

import com.robin.core.base.util.Const;
import com.robin.core.convert.util.ConvertUtil;
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
import com.robin.core.fileaccess.meta.DataCollectionMeta;
import com.robin.core.fileaccess.meta.DataSetColumnMeta;

Expand Down Expand Up @@ -44,7 +45,10 @@ public XmlFileIterator(){
public XmlFileIterator(DataCollectionMeta metaList) {
super(metaList);
identifier= Const.FILEFORMATSTR.XML.getValue();

}
public XmlFileIterator(DataCollectionMeta metaList, AbstractFileSystemAccessor accessor) {
super(metaList,accessor);
identifier= Const.FILEFORMATSTR.XML.getValue();
}
@Override
public void beforeProcess() {
Expand Down
5 changes: 5 additions & 0 deletions common/src/main/java/com/robin/core/fileaccess/writer/AbstractQueueWriter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import com.robin.core.base.exception.OperationNotSupportException;
import com.robin.core.base.util.Const;
import com.robin.core.base.util.ResourceConst;
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
import com.robin.core.fileaccess.meta.DataCollectionMeta;
import com.twitter.bijection.Injection;
import com.twitter.bijection.avro.GenericAvroCodecs;
Expand Down Expand Up @@ -86,4 +87,8 @@ public void setOutputStream(OutputStream outputStream){
throw new OperationNotSupportException("");
}

@Override
public void setAccessUtil(AbstractFileSystemAccessor accessUtil) {
throw new OperationNotSupportException("");
}
}
3 changes: 3 additions & 0 deletions common/src/main/java/com/robin/core/fileaccess/writer/IResourceWriter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
package com.robin.core.fileaccess.writer;

import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;

import javax.naming.OperationNotSupportedException;
import java.io.BufferedWriter;
import java.io.Closeable;
Expand All @@ -17,5 +19,6 @@ public interface IResourceWriter extends Closeable {
String getIdentifier();
void setWriter(BufferedWriter writer);
void setOutputStream(OutputStream outputStream);
void setAccessUtil(AbstractFileSystemAccessor accessUtil);

}
3 changes: 2 additions & 1 deletion common/src/main/java/com/robin/core/fileaccess/writer/TextFileWriterFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,8 @@ public static IResourceWriter getWriterByType(DataCollectionMeta colmeta, Abstra

Class<? extends IResourceWriter> writerClass=fileWriterMap.get(fileSuffix);
if (!ObjectUtils.isEmpty(writerClass)) {
fileWriter = writerClass.getConstructor(DataCollectionMeta.class,AbstractFileSystemAccessor.class).newInstance(colmeta,accessor);
fileWriter = writerClass.getConstructor(DataCollectionMeta.class).newInstance(colmeta);
fileWriter.setAccessUtil(accessor);
logger.info("using resource writer {}",writerClass.getCanonicalName());
}

Expand Down
7 changes: 7 additions & 0 deletions common/src/main/java/com/robin/core/resaccess/iterator/AbstractQueueIterator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
package com.robin.core.resaccess.iterator;

import com.robin.core.base.exception.OperationNotSupportException;
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
import com.robin.core.fileaccess.iterator.AbstractResIterator;
import com.robin.core.fileaccess.meta.DataCollectionMeta;
import com.robin.core.fileaccess.util.AvroUtils;
Expand Down Expand Up @@ -46,4 +48,9 @@ public Map<String, Object> next() {
return null;
}
public abstract List<Map<String,Object>> pollMessage() throws IOException;

@Override
public void setAccessUtil(AbstractFileSystemAccessor accessUtil) {
throw new OperationNotSupportException("");
}
}
6 changes: 6 additions & 0 deletions common/src/main/java/com/robin/core/resaccess/iterator/JdbcResIterator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,9 @@
import com.robin.comm.dal.holder.db.JdbcResourceHolder;
import com.robin.comm.dal.pool.ResourceAccessHolder;
import com.robin.core.base.dao.SimpleJdbcDao;
import com.robin.core.base.exception.OperationNotSupportException;
import com.robin.core.base.spring.SpringContextHolder;
import com.robin.core.fileaccess.fs.AbstractFileSystemAccessor;
import com.robin.core.fileaccess.iterator.AbstractResIterator;
import com.robin.core.fileaccess.meta.DataCollectionMeta;
import lombok.extern.slf4j.Slf4j;
Expand Down Expand Up @@ -107,4 +109,8 @@ public String getIdentifier() {
return "jdbc";
}

@Override
public void setAccessUtil(AbstractFileSystemAccessor accessUtil) {
throw new OperationNotSupportException("");
}
}
41 changes: 21 additions & 20 deletions core/src/main/java/com/robin/core/base/dao/JdbcDao.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -524,26 +524,24 @@ public <T extends BaseObject, P extends Serializable> P createVO(T obj, Class<P>
returnTemplate().update(factory.newPreparedStatementCreator(insertSegment.getParams()), keyHolder);
retval = keyHolder.getKey().longValue();

if (!ObjectUtils.isEmpty(retval)) {
//assign increment column
if (generateColumn != null) {
FieldContent pkColumn = AnnotationRetriever.getPrimaryField(fields);
if (pkColumn == null) {
throw new DAOException("model " + obj.getClass().getSimpleName() + " does not have primary key");
}
Object targetVal = ReflectUtils.getIncrementValueBySetMethod(generateColumn.getSetMethod(), retval);
if (pkColumn.getPrimaryKeys() == null) {
generateColumn.getSetMethod().invoke(obj, targetVal);
retObj = (P) targetVal;
} else {
for (FieldContent field : pkColumn.getPrimaryKeys()) {
if (field.isIncrement() || field.isSequential()) {
field.getSetMethod().invoke(generateColumn.getGetMethod().invoke(obj), retval);
}
if (!ObjectUtils.isEmpty(retval) && (generateColumn != null)) {
FieldContent pkColumn = AnnotationRetriever.getPrimaryField(fields);
if (pkColumn == null) {
throw new DAOException("model " + obj.getClass().getSimpleName() + " does not have primary key");
}
Object targetVal = ReflectUtils.getIncrementValueBySetMethod(generateColumn.getSetMethod(), retval);
if (pkColumn.getPrimaryKeys() == null) {
generateColumn.getSetMethod().invoke(obj, targetVal);
retObj = (P) targetVal;
} else {
for (FieldContent field : pkColumn.getPrimaryKeys()) {
if (field.isIncrement() || field.isSequential()) {
field.getSetMethod().invoke(generateColumn.getGetMethod().invoke(obj), retval);
}
retObj = (P) pkColumn.getGetMethod().invoke(obj);
}
retObj = (P) pkColumn.getGetMethod().invoke(obj);
}

}
} else {
//no pk model insert
Expand Down Expand Up @@ -811,15 +809,18 @@ private List<Map<String, Object>> queryItemList(final PageQuery<Map<String, Obje
start = (pageNum - 1) * pageSize;
end = pageNum * pageSize;
}
return this.returnTemplate().query(pageSQL, new SplitPageResultSetExtractor(start, end, lobHandler) {});
return this.returnTemplate().query(pageSQL, new SplitPageResultSetExtractor(start, end, lobHandler) {
});
}

private List<Map<String, Object>> queryAllItemList(final String querySQL, Object... obj) {
return this.returnTemplate().query(querySQL, obj, new SplitPageResultSetExtractor(0, 0, lobHandler) {});
return this.returnTemplate().query(querySQL, obj, new SplitPageResultSetExtractor(0, 0, lobHandler) {
});
}

private List<Map<String, Object>> queryAllItemList(final String querySQL, final List<FieldContent> mappingFieldList, Object[] obj) {
return this.returnTemplate().query(querySQL, obj, new SplitPageResultSetExtractor(0, 0, lobHandler, mappingFieldList) {});
return this.returnTemplate().query(querySQL, obj, new SplitPageResultSetExtractor(0, 0, lobHandler, mappingFieldList) {
});
}

private void generateQuerySqlBySingleFields(FieldContent columncfg, Const.OPERATOR oper, StringBuilder queryBuffer, int length) {
Expand Down
1 change: 0 additions & 1 deletion core/src/main/java/com/robin/core/base/service/IMybatisBaseService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@

import com.baomidou.mybatisplus.core.conditions.Wrapper;
import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;
import com.baomidou.mybatisplus.core.metadata.IPage;
import com.baomidou.mybatisplus.core.toolkit.support.SFunction;
import com.baomidou.mybatisplus.extension.service.IService;
Expand Down
Loading
Loading