Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip parameter not working in MongoTemplate#remove(Query, Class). [DATAMONGO-1870] #2771

Closed
spring-projects-issues opened this issue Feb 11, 2018 · 3 comments

Comments

@spring-projects-issues
Copy link

@spring-projects-issues spring-projects-issues commented Feb 11, 2018

Eugene Tenkaev opened DATAMONGO-1870 and commented

Next code will remove all entries without skipping first 10 entries:

Criteria criteria = 
	Criteria.where("key").is(key);
Query query =
    Query
        .query(criteria)
        .with(new Sort(Sort.Direction.DESC, "date"))
        .skip(10);
WriteResult writeResult =
    mongoTemplate.remove(query, Item.class);

However MongoTemplate.findAllAndRemove works as expected:

Criteria criteria = 
	Criteria.where("key").is(key);
Query query =
    Query
        .query(criteria)
        .with(new Sort(Sort.Direction.DESC, "date"))
        .skip(10);
List<Item> writeResult =
    mongoTemplate.findAllAndRemove(query, Item.class);

This code will leave first 10 entries, all other - deleted


Affects: 1.10.10 (Ingalls SR10)

Referenced from: pull request #531

Backported to: 2.0.4 (Kay SR4), 1.10.11 (Ingalls SR11)

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Feb 11, 2018

Eugene Tenkaev commented

For whose who faced with my problem, I use next solution, I solve by doing next:

    private void deleteStaleItems() {
        Criteria criteria = Criteria.where("key").is(key);
        Query query =
            Query
                .query(criteria)
                .with(new Sort(Sort.Direction.DESC, "date"))
                .skip(10);
        DBCollection collection =
            mongoTemplate.getCollection(mongoTemplate.getCollectionName(Item.class));
        try (DBCursor cursor =
                 collection
                     .find(query.getQueryObject())
                     .sort(query.getSortObject())
                     .skip(query.getSkip())) {
            LinkedList<String> idsToRemove = new LinkedList<>();
            while (cursor.hasNext()) {
                DBObject dbObject = cursor.next();
                String id = (String) dbObject.get("_id");
                idsToRemove.add(id);
                if (idsToRemove.size() >= 30) {
                    bulkRemove(collection, idsToRemove);
                    idsToRemove.clear();
                }
            }
            bulkRemove(collection, idsToRemove);
        }
    }

    private void bulkRemove(DBCollection collection, LinkedList<String> ids) {
        Criteria criteria = Criteria.where("_id").in(ids);
        Query query = Query.query(criteria);
        collection.remove(query.getQueryObject());
    }

This helps to remove all old items that not in list of 10 items.
Also I'm doing delete in bulk manner - every 30 items.

I'm also have question to devs: is this solution not consumes huge amount of RAM on big datasets(1kk+ items)?

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Feb 12, 2018

Christoph Strobl commented

Thank you Hronom.
I think we could definatly improve on the documentation side of MongoOperations clearly stating the difference between remove and findAllAndRemove. Additionally I'll investigate if we can be "smarter" when it comes to remove where the Query actually yields skip and limit.

Concerning memory consumtion - yes loading hughe amounts of Documents into memory can be costly. Restricting fields to just the _id may help reduce that burden

@spring-projects-issues
Copy link
Author

@spring-projects-issues spring-projects-issues commented Feb 13, 2018

Christoph Strobl commented

PR for 1.x - #531
PR for 2.x - #532

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants