Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Partitioner data lose - SqlPagingQueryProviderFactoryBean [BATCH-2309] #1294

Closed
spring-projects-issues opened this issue Oct 9, 2014 · 7 comments
Labels
in: core status: invalid type: bug
Milestone

Comments

@spring-projects-issues
Copy link
Collaborator

@spring-projects-issues spring-projects-issues commented Oct 9, 2014

Victor Jabur opened BATCH-2309 and commented

I created a project based on "Reference URL" mentioned above that seems to lose data. The configuration where the problem occurs is listed below:

grid-size=10
commit-interval=100
pageSize=1000

maxActive=11 (org.apache.commons.dbcp.BasicDataSource)
maxIdle=11 (org.apache.commons.dbcp.BasicDataSource)

My source table (oracle database 11g) has 10387 records, but when the job process, it only takes 10117 records to the target database (also oracle 11g).

To conclude, that are 270 records that weren't integrated.


Affects: 2.1.7, 3.0.1

Reference URL: https://github.com/spring-projects/spring-batch/blob/master/spring-batch-samples/src/main/resources/jobs/partitionJdbcJob.xml

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Oct 9, 2014

Michael Minella commented

Can you please provide your configuration or a sample project that illustrates the issue?

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Oct 9, 2014

Victor Jabur commented

Sample project is reference url, a sample from spring github

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Oct 9, 2014

Michael Minella commented

The reference URL points to our sample. Not the one with your configuration.

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Oct 9, 2014

Victor Jabur commented

I can try prepare a full example with database scripts and reproduce the error with this poc

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Oct 9, 2014

Victor Jabur commented

I will prepare tomorrow, i am at home now. Thanks

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Nov 11, 2014

Victor Jabur commented

This is not a bug, i was using the partition feature incorrectly. I create a post on my blog to explain my initial problem and help other people with the same problem. Resuming: I was using SqlPagingReader with Rownum to resolve the problem to divide same quantity of records to each thread. But the processing of all records is not guaranteed.

This is the post: http://wp.me/pVXBl-8R

Thank you for very fast treatment here in the Jira.

Victor Jabur

@spring-projects-issues
Copy link
Collaborator Author

@spring-projects-issues spring-projects-issues commented Nov 11, 2014

Michael Minella commented

Thanks for the update!

@spring-projects-issues spring-projects-issues added type: bug status: invalid in: core labels Dec 16, 2019
@spring-projects-issues spring-projects-issues added this to the 3.0.3 milestone Dec 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in: core status: invalid type: bug
Projects
None yet
Development

No branches or pull requests

1 participant