Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

String value of datasource name with blocking queue reader #193

Closed
Kmouille opened this Issue Jun 10, 2016 · 3 comments

Comments

2 participants
@Kmouille
Copy link

Kmouille commented Jun 10, 2016

Hi Mahmoud,

First of all, thanks for this project. I like it a lot.

Following principles of tutorial "3. Dispatching records", I found out that the data source name in the job parameter is set as a String when job launches.

In my case I have a master job and 2 sub jobs reading a blokingQueue. If I launch the 3 jobs with a thread pool size of 2 then, the 3rd job (subjob 2) launches with thousands of elements in the queue.

Unfortunately, the data source name is set there with the current content of the blocking queue. This lists all the elements contained in that queue (thousands). This is a big string to store, and printing it out in the reports is very hard (that's not a readable report anymore ;-) )
Maybe the size of the queue at the beginning of the job could be enough ? or the size and the first n elements...

@benas

This comment has been minimized.

Copy link
Contributor

benas commented Jun 12, 2016

Hi @Kmouille Thank you, I'm glad you like Easy Batch 😄

Thank you for pointing out this issue! Indeed, not very convenient to dump the queue content in the job report.. The BlockingQueueRecordReader uses the toString method as the data source name, which prints the queue content by default.

Maybe the size of the queue at the beginning of the job could be enough ? or the size and the first n elements...

The size of the queue would be redundant with the total records count in the job report. I suggest to simply put "In-Memory Queue" just like (and to be consistent with) the StringRecordReader (which avoids printing the whole string in the report). What do you think?

Kind regards
Mahmoud

@benas benas added the enhancement label Jun 12, 2016

@benas benas added this to the 4.1.0 milestone Jun 12, 2016

@benas benas self-assigned this Jun 12, 2016

benas added a commit that referenced this issue Jun 15, 2016

issue #193: fix data source name of the BlockingQueueRecordReader to …
…not dump the queue content in the job report
@benas

This comment has been minimized.

Copy link
Contributor

benas commented Jun 15, 2016

Hi,

I've updated the BlockingQueueRecordReader to print In-Memory Queue as data source name. This is to be consistent with the StringRecordReader and IterableRecordReader (which respectively print In-Memory String and In-Memory Iterable)

With this fix, elements in the queue will not be dumped in the job report.

Kind regards
Mahmoud

@benas

This comment has been minimized.

Copy link
Contributor

benas commented Jun 17, 2016

Hi @Kmouille

The fix has been merged in the master branch and will be released in the upcoming version soon.

Regards
Mahmoud

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.