Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

实现一个hive Reader插件 #201

Closed
baisui1981 opened this issue Mar 28, 2023 · 1 comment
Closed

实现一个hive Reader插件 #201

baisui1981 opened this issue Mar 28, 2023 · 1 comment
Labels
enhancement New feature or request
Milestone

Comments

@baisui1981
Copy link
Member

离线分析完成之后需要将离线分析结果导入到外部存储中,需要Hive DataX Reader插件,之前hdfs Reader虽然也能用,但是配置太复杂,需要借助Hive MetaData 数据来反射hive的表存储信息,自动生成col meta

@baisui1981 baisui1981 added enhancement New feature or request 3.8.0 labels Mar 28, 2023
@baisui1981 baisui1981 added this to the v3.8.0 milestone Apr 13, 2023
@baisui1981 baisui1981 modified the milestones: v3.8.0, v4.0.0 Jul 2, 2023
@baisui1981 baisui1981 removed the 3.8.0 label Jul 2, 2023
@baisui1981 baisui1981 modified the milestones: v4.0.0, v3.8.0 Sep 1, 2023
@baisui1981
Copy link
Member Author

baisui1981 commented Sep 1, 2023

已经支持,并且做了如下扩展:

  1. 伙伴插件工厂
    DataXReader使用com.qlangtech.tis.plugin.CompanionPluginFactory 为 SelectedTab 创建伙伴插件,为DataXReader的SelectedTab提供属性扩展。
    以下是在 DataXMongodbReader中使用的例子:

      public class DataXMongodbReader extends BasicDataXRdbmsReader<MangoDBDataSourceFactory> implements IMongoTableFinder {
         public static class DefaultDescriptor extends BasicDataXRdbmsReaderDescriptor //
               implements SubForm.ISubFormItemValidate, CompanionPluginFactory<SelectedTabExtend> {
    
           @Override
           public SelectedTabExtend getCompanionPlugin(UploadPluginMeta pluginMeta) {
               // ...
               return SelectedTabExtend.getBatchPluginStore(pluginMeta.getPluginContext(),
                       pluginMeta.getDataXName(true)).find(tabName, false);
           }
    
           @Override
           public Descriptor<SelectedTabExtend> getCompanionDescriptor() {
               return TIS.get().getDescriptor(MongoSelectedTabExtend.class);
           }
       }
    }
  2. 可以在不对SelectedTab 继承的情况下对DataXHiveReader.selectedTabs 的列表实例(SelectedTab) 进行扩展,使用com/qlangtech/tis/hive/reader/DataXHiveReader.selectedTabs.json配置文件 ,改变默认cols的显示方式 com.qlangtech.tis.extension.util.PluginExtraProps.ViewType 的 idlist 进行修改。

    {
    "cols": {
      "viewtype": "tuplelist", //改变成table方式显示
      "elementCreator": "com.qlangtech.tis.plugin.datax.mongo.MongoCMetaCreatorFactory",
      }
    }

    该配置文件是通过

    Optional<PluginExtraProps> overwriteSubform = PluginExtraProps.parseExtraProps(parentClazz,
    Optional.of(subFormField));
    overwriteSubform.ifPresent((ep) -> {
    ep.forEach((fieldKey, val) -> {
    Objects.requireNonNull(fieldsType.get(fieldKey),
    "fieldKey:" + fieldKey + " relevant PropertyType " + "can" + " not be null").extraProp.merge(val);
    });
    代码覆写的

配置参数:

viewtype

idlist tuplelist
截屏2023-09-01 12 05 26 截屏2023-09-01 12 03 13

elementCreator

source 自定义扩展 element(CMeta)创建方法:

    public interface ElementCreatorFactory{
        CMeta create(JSONObject targetCol);
    }

源端 Selected Tab 校验:

由于dataX reader 中不能扩展 selectedTab,然后又通过自定义elementCreator属性修改了 selectedTable中cols中的Element元素类型,需要在selectedTab 表单提交的时,对表单进行校验,所以引入:

可以让selectTab 在提交时由宿主DataX Reader 执行表单校验:

interface ISubFormItemValidate {
        /**
         * 校验全部 selected tabs 表单
         *
         * @param msgHandler
         * @param context
         * @param props
         * @param subFormFilter
         * @param formData
         * @return
         */
        boolean validateSubFormItems(IControlMsgHandler msgHandler, Context context, BaseSubFormProperties props,
                                     IPropertyType.SubFormFilter subFormFilter, AttrVals formData);

        /**
         * 校验单个selected表单
         *
         * @param msgHandler
         * @param context
         * @param tab
         * @return
         */
        boolean validateSubForm(IControlMsgHandler msgHandler, Context context, SelectedTab tab);
    }

@baisui1981 baisui1981 changed the title 需要一个hive Reader插件 实现一个hive Reader插件 Sep 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant