Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve logging #218

Open
yeikel opened this issue Jan 10, 2020 · 2 comments
Open

Improve logging #218

yeikel opened this issue Jan 10, 2020 · 2 comments
Assignees
@zouzias

Comments

@yeikel
Copy link
Contributor

yeikel commented Jan 10, 2020 •
edited

Is your feature request related to a problem? Please describe.
I would be nice to have more logs to understand what is going on with the application. Sometimes it seems stuck.

Even with the log level set to DEBUG there is not a lot of information to trace the progress. For example, consider the following logging configuration :

log4j.logger.org.zouzias=DEBUG, console

And the log messages :

20/01/10 17:03:23 INFO LuceneRDD: Instance is created...
20/01/10 17:03:23 INFO LuceneRDD: Instance is created...
20/01/10 17:03:23 INFO LuceneRDD: Linkage requested
20/01/10 17:03:23 INFO LuceneRDD: Linkage requested
20/01/10 17:03:23 INFO LuceneRDD: Linker method is cartesian
20/01/10 17:03:23 INFO LuceneRDD: Linker method is cartesian
20/01/10 17:03:23 INFO LuceneRDD: Computing top-k linkage per partition
20/01/10 17:03:23 INFO LuceneRDD: Computing top-k linkage per partition
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=5] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=5] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Config parameter lucenerdd.index.store.mode is set to 'disk'
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Lucene index will be storage in disk
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: Index disk location /var/folders/0l/r325x8xd7td1qvw_04wjn15r0000gn/T/
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9] Partition is created...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3]Indexing process initiated at 2020-01-10T17:03:24.580-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3]Indexing process initiated at 2020-01-10T17:03:24.580-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9]Indexing process initiated at 2020-01-10T17:03:24.582-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9]Indexing process initiated at 2020-01-10T17:03:24.582-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=5]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=5]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0]Indexing process initiated at 2020-01-10T17:03:24.581-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0]Indexing process initiated at 2020-01-10T17:03:24.581-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11]Indexing process initiated at 2020-01-10T17:03:24.579-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10]Indexing process completed at 2020-01-10T17:03:24.626-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10]Indexing process completed at 2020-01-10T17:03:24.626-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1]Indexing process completed at 2020-01-10T17:03:24.627-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11]Indexing process completed at 2020-01-10T17:03:24.626-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11]Indexing process completed at 2020-01-10T17:03:24.626-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=9]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=0]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=11]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=4]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=7]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=2]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=10]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=1]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=6]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=3]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8]Indexing process completed at 2020-01-10T17:03:24.628-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8]Indexing process completed at 2020-01-10T17:03:24.628-05:00...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8]Indexing process took 0 seconds...
20/01/10 17:03:24 INFO LuceneRDDPartition: [partId=8]Indexing process took 0 seconds...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=1]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=1]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=8]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=8]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=4]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=4]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=0]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=0]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=11]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=11]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=6]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=6]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=2]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=2]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=9]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=9]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=10]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=10]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=7]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=7]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=3]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=3]Closing index writers...
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=0]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=0]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=7]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=7]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=11]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=11]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=9]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=9]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=6]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=6]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=4]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=4]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=2]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=2]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=10]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=10]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=8]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=8]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=1]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=1]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=3]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=3]Instantiating index/facet readers
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=6]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=6]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=0]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=0]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=7]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=7]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=1]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=1]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=3]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=3]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=8]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=8]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=2]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=2]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=9]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=9]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=11]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=11]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=10]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=10]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=4]Index readers instantiated successfully
20/01/10 17:03:25 DEBUG LuceneRDDPartition: [partId=4]Index readers instantiated successfully
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=7]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=7]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=0]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=0]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=6]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=6]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=4]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=4]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=11]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=11]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=10]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=10]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=9]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=9]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=2]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=2]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=1]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=1]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=8]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=8]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=3]Indexed 0 documents
20/01/10 17:03:25 INFO LuceneRDDPartition: [partId=3]Indexed 0 documents
20/01/10 17:03:46 INFO LuceneRDDPartition: [partId=5]Indexing process completed at 2020-01-10T17:03:46.639-05:00...
20/01/10 17:03:46 INFO LuceneRDDPartition: [partId=5]Indexing process completed at 2020-01-10T17:03:46.639-05:00...
20/01/10 17:03:46 INFO LuceneRDDPartition: [partId=5]Indexing process took 22 seconds...
20/01/10 17:03:46 INFO LuceneRDDPartition: [partId=5]Indexing process took 22 seconds...
20/01/10 17:03:49 DEBUG LuceneRDDPartition: [partId=5]Closing index writers...
20/01/10 17:03:49 DEBUG LuceneRDDPartition: [partId=5]Closing index writers...
20/01/10 17:03:49 DEBUG LuceneRDDPartition: [partId=5]Instantiating index/facet readers
20/01/10 17:03:49 DEBUG LuceneRDDPartition: [partId=5]Instantiating index/facet readers
20/01/10 17:03:49 DEBUG LuceneRDDPartition: [partId=5]Index readers instantiated successfully
20/01/10 17:03:49 DEBUG LuceneRDDPartition: [partId=5]Index readers instantiated successfully
20/01/10 17:03:49 INFO LuceneRDDPartition: [partId=5]Indexed 897134 documents
20/01/10 17:03:49 INFO LuceneRDDPartition: [partId=5]Indexed 897134 documents

After the documents , are indexed , there is no information about what is happening on the cluster/process. I'd be nice to be able to trace the process even deeper
@zouzias zouzias self-assigned this Jan 13, 2020
@zouzias
Copy link
Owner

zouzias commented Jan 13, 2020 •
edited

How about an info message when the linkage action takes place since linkage can take a lot of time?

I just saw that you are using the "cartesian" linkage method. Does the broadcastjoin approach does not work for you? Cartesian could be very slow if you don't have a lot of resources.

20/01/10 17:03:23 INFO LuceneRDD: Linker method is cartesian

This is related to #190 .

@yeikel
Copy link
Contributor Author

yeikel commented Jan 13, 2020 •
edited

How about an info message when the linkage action takes place since linkage can take a lot of time?

Could you please clarify this?

I just saw that you are using the "cartesian" linkage method. Does the broadcastjoin approach does not work for you? Cartesian could be very slow if you don't have a lot of resources.

Sadly I can't do this as I need to use the Query Builders with the "link" method and it is not supported as we discussed in pull request #162

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 in stage 2.0 (TID 13) had a not serializable result: org.apache.lucene.search.BooleanQuery
Serialization stack:
	- object not serializable (class: org.apache.lucene.search.BooleanQuery, value: (home_ad:flt home_ad:blk home_ad:fullview home_ad:gdn home_ad:siu home_ad:sai home_ad:wan)~5)
	- field (class: scala.Tuple2, name: _2, type: class java.lang.Object)
	- object (class scala.Tuple2, (0,(home_ad:flt home_ad:blk home_ad:fullview home_ad:gdn home_ad:siu home_ad:sai home_ad:wan)~5))
	- element of array (index: 0)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zouzias zouzias

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zouzias @yeikel