Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ServuceImpl的saveOrUpdateBatch中校验是否需要更新是单条查询造成速度慢的优化 #6229

Closed
sgps000 opened this issue Jun 1, 2024 · 1 comment

Comments

@sgps000
Copy link

sgps000 commented Jun 1, 2024

请详细描述需要增加的功能
版本:3.5.4.1
环境:JDK17

image

想要类似以下的优化实现,将单条校验改为批量查询校验:

public boolean saveOrUpdateBatch(Collection<T> entityList, int batchSize) {
        if (CollectionUtils.isEmpty(entityList)) return false;
        TableInfo tableInfo = TableInfoHelper.getTableInfo(entityClass);
        Assert.notNull(tableInfo, "error: can not execute. because can not find cache of TableInfo for entity!");
        String keyProperty = tableInfo.getKeyProperty();
        Assert.notEmpty(keyProperty, "error: can not execute. because can not find column for id from entity!");

        String sqlStatement = SqlHelper.getSqlStatement(this.mapperClass, SqlMethod.INSERT_ONE);
        Consumer<SqlSession> consumer = sqlSession -> {
            int size = entityList.size();
            int idxLimit = Math.min(batchSize, size);
            int i = 1;
            // 批处理临时集合
            List<T> batchList = new ArrayList<>();

            for (T element : entityList) {
                batchList.add(element);

                // 批处理
                if (i == idxLimit) {
                    // 批数据id集合
                    List<Object> idList = batchList.stream().map(e -> tableInfo.getPropertyValue(e, keyProperty))
                            .filter(StringUtils::checkValNotNull).toList();

                    // 查询批数据中是更新的id集合
                    List<Object> updateIdList = new ArrayList<>();
                    if (!idList.isEmpty()) {
                        // todo 此处写法不对,暂未找到
                        List<Object> tempList = sqlSession.selectList(getSqlStatement(SqlMethod.SELECT_BATCH_BY_IDS), idList);
                        updateIdList = tempList.stream().map(e -> tableInfo.getPropertyValue(e, keyProperty)).toList();
                    }

                    // 遍历批数据
                    for (T entity : batchList) {
                        Object id = tableInfo.getPropertyValue(entity, keyProperty);
                        if (StringUtils.checkValNull(id) || !updateIdList.contains(id)) {
                            sqlSession.insert(sqlStatement, entity);
                        } else {
                            MapperMethod.ParamMap<T> param = new MapperMethod.ParamMap<>();
                            param.put(Constants.ENTITY, entity);
                            sqlSession.update(getSqlStatement(SqlMethod.UPDATE_BY_ID), param);
                        }
                    }

                    sqlSession.flushStatements();
                    idxLimit = Math.min(idxLimit + batchSize, size);
                    batchList.clear();
                }
                i++;
            }
        };
        return SqlHelper.executeBatch(getSqlSessionFactory(), log, consumer);
    }

但是!

image

此处仅需要查询出所有已存在的id,不应该查全量数据,感觉这会浪费很多资源去传输、处理、存储,造成时间延长。能不能加一个仅查询某个字段的方法,业务中也非常实用

@nieqiurong
Copy link
Contributor

String sqlStatement = getSqlStatement(SqlMethod.SELECT_OBJS);
Map<String,Object> params = new HashMap<>();
params.put(Constants.WRAPPER, Wrappers.query().select(keyProperty).in(keyProperty, List.of(1,2,3)));
List<主键类型> objects = sqlSession.selectList(sqlStatement, params);
String sqlStatement = getSqlStatement(SqlMethod.SELECT_LIST);
Map<String,Object> params = new HashMap<>();
params.put(Constants.WRAPPER, Wrappers.query().select(keyProperty).in(keyProperty, List.of(1,2,3)));
List<泛型实体> objects = sqlSession.selectList(sqlStatement, params);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants