New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fast search #1100

Merged
merged 13 commits into from Apr 6, 2016

Conversation

Projects
None yet
7 participants
@simonharrer
Contributor

simonharrer commented Apr 4, 2016

  • Fix performance bug which caused the UI to slow down extremely

Search is now blazingly fast (compared to before)

  • New class MainTableDataModel which holds all sorted and filtered lists plus sorting and filtering logic.
  • If a search is still active while entering a new search term, the previous search is cancelled. This improves search time quite a lot.
  • By filtering after the sorting, the search is extremely fast as the amount of sorting is reduced dramatically
  • Using "filter" search should be now the fastest search, as no sorting has to take place

Issues

  • bugs using the float search (but filter works as expected)
  • remove sysout statement
  • tests fail as the mode of the database is written back directly to the file

Please try this out.

Related

  • #1094 as this improves performance of the main table (but without numbers)
  • #1034 may no longer be required to be fixed in #1037

simonharrer added some commits Apr 4, 2016

Search is now blazingly fast.
- New class MainTableDataModel which holds all sorted and filtered lists plus sorting and filtering logic.
- If a search is still active while entering a new search term, the previous search is cancelled. This improves search time quite a lot.
- By filtering after the sorting, the search is extremely fast as the amount of sorting is reduced dramatically
- Using "filter" search should be now the fastest search, as no sorting has to take place
Fix another performance bug which now allows to load bib databases ve…
…ry quickly (5 sec vs. 60 sec for 40k entries)

@stefan-kolb stefan-kolb added this to the v3.3 milestone Apr 5, 2016

@stefan-kolb

This comment has been minimized.

Show comment
Hide comment
@stefan-kolb

stefan-kolb Apr 5, 2016

Member

L(fast)TM 👍

Member

stefan-kolb commented Apr 5, 2016

L(fast)TM 👍

@tobiasdiez tobiasdiez referenced this pull request Apr 5, 2016

Merged

Add benchmarks #1103

0 of 3 tasks complete
@tobiasdiez

This comment has been minimized.

Show comment
Hide comment
@tobiasdiez

tobiasdiez Apr 5, 2016

Member

LGTM 👍

Member

tobiasdiez commented Apr 5, 2016

LGTM 👍

simonharrer added some commits Apr 5, 2016

@tobiasdiez

This comment has been minimized.

Show comment
Hide comment
@tobiasdiez

tobiasdiez Apr 5, 2016

Member

I really don't like this...I suspect moving YearUtil to logic is also not possible, right?

I really don't like this...I suspect moving YearUtil to logic is also not possible, right?

This comment has been minimized.

Show comment
Hide comment
@simonharrer

simonharrer Apr 5, 2016

Contributor

Nope, not possible. But I see no other solution except not to make this improvement.

Contributor

simonharrer replied Apr 5, 2016

Nope, not possible. But I see no other solution except not to make this improvement.

This comment has been minimized.

Show comment
Hide comment
@tobiasdiez

tobiasdiez Apr 5, 2016

Member

Remove the distinction between model and logic? But I'm probably the only one which does not likes the model package...

Member

tobiasdiez replied Apr 5, 2016

Remove the distinction between model and logic? But I'm probably the only one which does not likes the model package...

This comment has been minimized.

Show comment
Hide comment
@simonharrer

simonharrer Apr 5, 2016

Contributor

Even if we would combine model and logic, we would have very bad dependency relations. By separating model and logic, they are simply revealed there.

Contributor

simonharrer replied Apr 5, 2016

Even if we would combine model and logic, we would have very bad dependency relations. By separating model and logic, they are simply revealed there.

@Siedlerchr

This comment has been minimized.

Show comment
Hide comment
@Siedlerchr

Siedlerchr Apr 5, 2016

Contributor

I really don't like the YearUtil class at all and I think it is obsolete, or at least its self created logic is obsolete.
As with the new Java 8 Release, there haven been important changes made to the Date-Logic.
Especially the new DateFormatters. Why don't use them instead of manually trying to do Date Arithmetic`?
http://www.tutorialspoint.com/java8/java8_datetime_api.htm
And here:
https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#ofPattern-java.lang.String-

Contributor

Siedlerchr commented on e0380b7 Apr 5, 2016

I really don't like the YearUtil class at all and I think it is obsolete, or at least its self created logic is obsolete.
As with the new Java 8 Release, there haven been important changes made to the Date-Logic.
Especially the new DateFormatters. Why don't use them instead of manually trying to do Date Arithmetic`?
http://www.tutorialspoint.com/java8/java8_datetime_api.htm
And here:
https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#ofPattern-java.lang.String-

This comment has been minimized.

Show comment
Hide comment
@simonharrer

simonharrer Apr 5, 2016

Contributor

I think this Year arithmetic cannot be easily handled with Java 8 classes, but you are welcome to try it.

Contributor

simonharrer replied Apr 5, 2016

I think this Year arithmetic cannot be easily handled with Java 8 classes, but you are welcome to try it.

This comment has been minimized.

Show comment
Hide comment
@Siedlerchr

Siedlerchr Apr 5, 2016

Contributor

I will look into that.

Contributor

Siedlerchr replied Apr 5, 2016

I will look into that.

@simonharrer simonharrer merged commit 756be32 into master Apr 6, 2016

3 of 6 checks passed

codacy/pr Not so good... This commit quality could be better.
Details
codecov/project 23.60% (-0.01%) compared to a97c720 at 23.61%
Details
coverage/coveralls Coverage decreased (-0.002%) to 27.276%
Details
ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@simonharrer simonharrer deleted the fast-search branch Apr 6, 2016

@tobiasdiez

This comment has been minimized.

Show comment
Hide comment
@tobiasdiez

tobiasdiez Apr 6, 2016

Member

Running the benchmarks with this PR merged in shows that the writing speed was improved by an additional factor of 10 (being now at roughly 60k entries per second). Good job 👍

Member

tobiasdiez commented Apr 6, 2016

Running the benchmarks with this PR merged in shows that the writing speed was improved by an additional factor of 10 (being now at roughly 60k entries per second). Good job 👍

@mlep

This comment has been minimized.

Show comment
Hide comment
@mlep

mlep Apr 6, 2016

Contributor

Considering users reported JabRef to be slow, this is a great improvement (worth being advertised in the blog!).

Contributor

mlep commented Apr 6, 2016

Considering users reported JabRef to be slow, this is a great improvement (worth being advertised in the blog!).

@simonharrer

This comment has been minimized.

Show comment
Hide comment
@simonharrer

simonharrer Apr 6, 2016

Contributor

The problem is, I think currently we lack people writing these blog posts... :)

Contributor

simonharrer commented Apr 6, 2016

The problem is, I think currently we lack people writing these blog posts... :)

@mlep

This comment has been minimized.

Show comment
Hide comment
@mlep

mlep Apr 6, 2016

Contributor

@simonharrer: A problem? Here is the stub of a solution.
I collated various information from different issues. I am not sure I got everything right, hence, @JabRef/developers, feel free to improve it!
Note: I think we should cite the names of people doing the work (code does not come out of the blue, right?), and invite users to give us feedback.

For later on: who is in charge of the blog?

%%%%%%%%%%%%%%%%%%%%%%%%%%

A faster JabRef is coming!

Some users reported JabRef was slow on large databases (thank you for the feedback!). This was especially the case for three operations:

  • loading a database
  • saving a database
  • searching through a database.

During a search, the user interface could become very unresponsive, which is indeed quite annoying... Well, this time will be over soon: Programmer @simonharrer has recently contributed code making search much faster (#1100).

Preliminary tests have been carried out on a database with 100000 entries (is it big enough for you?). They shows that JabRef is now 10 times faster at searching: about 60000 entries are searched per second.

If you want to give a try: http://builds.jabref.org/master/ (this is a development version. So, be carefully, and back up your data!). Your comments are welcomed at #1100

About opening large databases: work is under way to speed it up You can follow this development at #1094

Contributor

mlep commented Apr 6, 2016

@simonharrer: A problem? Here is the stub of a solution.
I collated various information from different issues. I am not sure I got everything right, hence, @JabRef/developers, feel free to improve it!
Note: I think we should cite the names of people doing the work (code does not come out of the blue, right?), and invite users to give us feedback.

For later on: who is in charge of the blog?

%%%%%%%%%%%%%%%%%%%%%%%%%%

A faster JabRef is coming!

Some users reported JabRef was slow on large databases (thank you for the feedback!). This was especially the case for three operations:

  • loading a database
  • saving a database
  • searching through a database.

During a search, the user interface could become very unresponsive, which is indeed quite annoying... Well, this time will be over soon: Programmer @simonharrer has recently contributed code making search much faster (#1100).

Preliminary tests have been carried out on a database with 100000 entries (is it big enough for you?). They shows that JabRef is now 10 times faster at searching: about 60000 entries are searched per second.

If you want to give a try: http://builds.jabref.org/master/ (this is a development version. So, be carefully, and back up your data!). Your comments are welcomed at #1100

About opening large databases: work is under way to speed it up You can follow this development at #1094

@stefan-kolb

This comment has been minimized.

Show comment
Hide comment
@stefan-kolb

stefan-kolb Apr 6, 2016

Member

Great, now we need the blog of the Stupro project so the students and alsomlep and other can add small blog posts!

Member

stefan-kolb commented Apr 6, 2016

Great, now we need the blog of the Stupro project so the students and alsomlep and other can add small blog posts!

@simonharrer

This comment has been minimized.

Show comment
Hide comment
@simonharrer

simonharrer Apr 6, 2016

Contributor

the blog is already online, I think.

Contributor

simonharrer commented Apr 6, 2016

the blog is already online, I think.

@mlep

This comment has been minimized.

Show comment
Hide comment
@mlep

mlep Apr 6, 2016

Contributor

It is here: http://www.jabref.org/blog/
For a new post, one has simply to add a new .md file with the proper header to https://github.com/JabRef/www.jabref.org/tree/gh-pages/_posts, right?

Contributor

mlep commented Apr 6, 2016

It is here: http://www.jabref.org/blog/
For a new post, one has simply to add a new .md file with the proper header to https://github.com/JabRef/www.jabref.org/tree/gh-pages/_posts, right?

@lc9275

This comment has been minimized.

Show comment
Hide comment
@lc9275

lc9275 Apr 6, 2016

@mlep
I have a database of about 1300 articles. What is really slow with Jabref is scrolling (using either the wheel or dragging the scrollbar) - it is just never smooth.

lc9275 commented Apr 6, 2016

@mlep
I have a database of about 1300 articles. What is really slow with Jabref is scrolling (using either the wheel or dragging the scrollbar) - it is just never smooth.

@mlep

This comment has been minimized.

Show comment
Hide comment
@mlep

mlep Apr 6, 2016

Contributor

@lc9275 I have a database the same as as yours, and scrolling is ok.
First of all, could you confirm scrolling is slow for you when using the last development version (on a copy of your database): http://builds.jabref.org/master/

Contributor

mlep commented Apr 6, 2016

@lc9275 I have a database the same as as yours, and scrolling is ok.
First of all, could you confirm scrolling is slow for you when using the last development version (on a copy of your database): http://builds.jabref.org/master/

@lc9275

This comment has been minimized.

Show comment
Hide comment
@lc9275

lc9275 Apr 6, 2016

@mlep That solved the problem, nice and smooth! I was using a dev version of a month or two ago. Thanks for the quick feedback, and I am happy it is solved in the current version.

lc9275 commented Apr 6, 2016

@mlep That solved the problem, nice and smooth! I was using a dev version of a month or two ago. Thanks for the quick feedback, and I am happy it is solved in the current version.

@simonharrer

This comment has been minimized.

Show comment
Hide comment
Contributor

simonharrer commented Apr 7, 2016

@mlep mlep referenced this pull request Apr 7, 2016

Closed

[Blog's post] A faster JabRef #35

@oscargus

This comment has been minimized.

Show comment
Hide comment
@oscargus

oscargus Jul 26, 2016

Contributor

@simonharrer Passing null here seems to be a bad idea since the matcher is directly dereferences in matches. Or am I (and Find bugs) missing something?

@simonharrer Passing null here seems to be a bad idea since the matcher is directly dereferences in matches. Or am I (and Find bugs) missing something?

This comment has been minimized.

Show comment
Hide comment
@oscargus

oscargus Jul 26, 2016

Contributor

Same three line above.

Contributor

oscargus replied Jul 26, 2016

Same three line above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment