Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Override write(byte[] b, int off, int len) in FilterOutputStream for better performance #8749

Merged
merged 1 commit into from Dec 2, 2014

Conversation

s1monw
Copy link
Contributor

@s1monw s1monw commented Dec 2, 2014

Closes #8748

@rmuir
Copy link
Contributor

rmuir commented Dec 2, 2014

+1, this is a horrible trap. It has a real impact on write performance. Lucene tests got like 8x slower because of one of these that was just being used to do something on close()

@s1monw
Copy link
Contributor Author

s1monw commented Dec 2, 2014

thx for the review @rmuir

@s1monw s1monw merged commit d8f1617 into elastic:master Dec 2, 2014
@s1monw s1monw deleted the issues/8748 branch December 2, 2014 20:29
@clintongormley clintongormley changed the title Override write(byte[] b, int off, int len) in FilterOutputStream for better performance Snapshot/Restore: Override write(byte[] b, int off, int len) in FilterOutputStream for better performance Dec 16, 2014
@agallou
Copy link

agallou commented Jan 26, 2015

Thanks for the fix.

We were using the 1.4.1 version and the snapshot took 5 hours (for 8.52Gi and 1 801 986 docs).
The same snapshot in version 1.4.2 now takes less than one minute.

Snapshot in version 1.4.1 :

{
    "snapshot" : "bkp_20150122214749",
    "indices" : [ "livre.20150122211008", "editeur.20150122211008", "auteur.20150122211008", "fournisseur.20150122211008" ],
    "state" : "SUCCESS",
    "start_time" : "2015-01-22T20:47:49.941Z",
    "start_time_in_millis" : 1421959669941,
    "end_time" : "2015-01-23T01:52:00.983Z",
    "end_time_in_millis" : 1421977920983,
    "duration_in_millis" : 18251042,
    "failures" : [ ],
    "shards" : {
      "total" : 20,
      "failed" : 0,
      "successful" : 20
    }
}

Snapshot in version 1.4.2 :

{
    "snapshot" : "bkp_20150126220702",
    "indices" : [ "livre.20150126002727", "editeur.20150126002727", "fournisseur.20150126002727", "auteur.20150126002727" ],
    "state" : "SUCCESS",
    "start_time" : "2015-01-26T21:07:02.522Z",
    "start_time_in_millis" : 1422306422522,
    "end_time" : "2015-01-26T21:07:42.867Z",
    "end_time_in_millis" : 1422306462867,
    "duration_in_millis" : 40345,
    "failures" : [ ],
    "shards" : {
      "total" : 20,
      "failed" : 0,
      "successful" : 20
    }
}

@clintongormley
Copy link

awesome - thanks for the feedback @agallou

@clintongormley clintongormley added the :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs label Mar 19, 2015
@clintongormley clintongormley changed the title Snapshot/Restore: Override write(byte[] b, int off, int len) in FilterOutputStream for better performance Override write(byte[] b, int off, int len) in FilterOutputStream for better performance Jun 7, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Snapshot/Restore: FilterOutputStream in FsBlobContainer writes every individual byte
4 participants