Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use tokenizer for extraction; add benchmark #424

Merged
merged 77 commits into from
Dec 16, 2021
Merged

Use tokenizer for extraction; add benchmark #424

merged 77 commits into from
Dec 16, 2021

Commits on Sep 13, 2021

  1. Move to from vec to streams 

    Previously we collected all inputs in one vector
before checking the links, which is not ideal.
Especially when reading many inputs (e.g. by using a glob pattern),
this could cause issues like running out of file handles.

By moving to streams we avoid that scenario. This is also the first
step towards improving performance for many inputs.
    @mre
    mre committed Sep 13, 2021
    Configuration menu
    Copy the full SHA
    613bf5f View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of github.com:lycheeverse/lychee into stream

    @mre
    mre committed Sep 13, 2021
    Configuration menu
    Copy the full SHA
    aee5c9f View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2021

  1. Fix formatting and lints

    @mre
    mre committed Sep 14, 2021
    Configuration menu
    Copy the full SHA
    0c05acb View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2021

  1. Merge branch 'master' of github.com:lycheeverse/lychee into stream

    @mre
    mre committed Sep 20, 2021
    Configuration menu
    Copy the full SHA
    eecd8e2 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2021

  1. Merge remote-tracking branch 'upstream/master' into stream

    Timo Freiberg committed Sep 26, 2021
    Configuration menu
    Copy the full SHA
    18448ce View commit details
    Browse the repository at this point in the history

  2. Return collected links as Stream

    Timo Freiberg committed Sep 26, 2021
    Configuration menu
    Copy the full SHA
    a05400b View commit details
    Browse the repository at this point in the history

Commits on Oct 6, 2021

  1. Initialize ProgressBar without length 

    Because we can't know the amount of links without blocking
    Timo Freiberg committed Oct 6, 2021
    Configuration menu
    Copy the full SHA
    d4b9bad View commit details
    Browse the repository at this point in the history

  2. Handle stream results in main thread, not in task 

    To stay as close to the pre-stream behaviour, we want to stop processing
as soon as an Err value appears in the stream. This is easiest when the
stream is consumed in the main thread.
Previously, the stream was consumed in a tokio task and the main thread
waited for responses.
Now, a tokio task waits for responses (and displays them/registers
response stats) and the main thread sends links to the ClientPool.
To ensure that the main thread waits for all responses to have arrived
before finishing the ProgressBar and printing the stats, it waits for
the show_results_task to finish.
    Timo Freiberg committed Oct 6, 2021
    Configuration menu
    Copy the full SHA
    98cdfba View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2021

  1. Merge branch 'master' of github.com:lycheeverse/lychee into stream

    @mre
    mre committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    23df173 View commit details
    Browse the repository at this point in the history

  2. Cleanup

    @mre
    mre committed Oct 7, 2021
    Configuration menu
    Copy the full SHA
    1471725 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2021

  1. Merge branch 'master' of github.com:lycheeverse/lychee into stream

    @mre
    mre committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    d111c0e View commit details
    Browse the repository at this point in the history

  2. Add basic directory support

    @mre
    mre committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    dfd0735 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' of github.com:lycheeverse/lychee into stream

    @mre
    mre committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    9ef2b7d View commit details
    Browse the repository at this point in the history

  4. Fix deadlock

    @mre
    Timo Freiberg authored and mre committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    5f790bf View commit details
    Browse the repository at this point in the history

  5. Clippy

    @mre
    mre committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    d42bf3e View commit details
    Browse the repository at this point in the history

  6. Add test for http protocol file type

    @mre
    mre committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    5bea0c8 View commit details
    Browse the repository at this point in the history

  7. Remove deadpool (once again) 

    Replaced with `futures::StreamExt::for_each_concurrent`.
    @mre
    mre committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    8ea4de6 View commit details
    Browse the repository at this point in the history

  8. Refactor main; fix tests

    @mre
    mre committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    f33468e View commit details
    Browse the repository at this point in the history

  9. Move commands into separate submodule

    @mre
    mre committed Oct 10, 2021
    Configuration menu
    Copy the full SHA
    1bfeb0e View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2021

  1. Reintegrate changes from master

    @mre
    mre committed Nov 24, 2021
    Configuration menu
    Copy the full SHA
    fb2dde2 View commit details
    Browse the repository at this point in the history

Commits on Nov 26, 2021

  1. Simplify input handling

    @mre
    mre committed Nov 26, 2021
    Configuration menu
    Copy the full SHA
    fec6f8f View commit details
    Browse the repository at this point in the history

  2. Simplify collector

    @mre
    mre committed Nov 26, 2021
    Configuration menu
    Copy the full SHA
    a69ea63 View commit details
    Browse the repository at this point in the history

  3. Remove unnecessary unwrap

    @mre
    mre committed Nov 26, 2021
    Configuration menu
    Copy the full SHA
    c83429c View commit details
    Browse the repository at this point in the history

  4. Simplify main

    @mre
    mre committed Nov 26, 2021
    Configuration menu
    Copy the full SHA
    b231175 View commit details
    Browse the repository at this point in the history

  5. cleanup check

    @mre
    mre committed Nov 26, 2021
    Configuration menu
    Copy the full SHA
    1d80866 View commit details
    Browse the repository at this point in the history

  6. clean up dump command

    @mre
    mre committed Nov 26, 2021
    Configuration menu
    Copy the full SHA
    ee4dd9c View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2021

  1. Move to String, which is Send 

    Tendril is not Send by default
    @mre
    mre committed Nov 28, 2021
    Configuration menu
    Copy the full SHA
    52e52bf View commit details
    Browse the repository at this point in the history

  2. Revert "Move to String, which is Send" 

    This reverts commit 52e52bf.
    @mre
    mre committed Nov 28, 2021
    Configuration menu
    Copy the full SHA
    89d7566 View commit details
    Browse the repository at this point in the history

  3. Revert "Revert "Move to String, which is Send"" 

    This reverts commit 89d7566.
    @mre
    mre committed Nov 28, 2021
    Configuration menu
    Copy the full SHA
    faf40a8 View commit details
    Browse the repository at this point in the history

  4. Parallel stream awesomeness

    @mre
    mre committed Nov 28, 2021
    Configuration menu
    Copy the full SHA
    63a8370 View commit details
    Browse the repository at this point in the history

  5. cleanup

    @mre
    mre committed Nov 28, 2021
    Configuration menu
    Copy the full SHA
    562f112 View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2021

  1. Add back Result

    @mre
    mre committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    e57bd6c View commit details
    Browse the repository at this point in the history

  2. fmt

    @mre
    mre committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    954ba0d View commit details
    Browse the repository at this point in the history

  3. Fix wording in test

    @mre
    mre committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    7945421 View commit details
    Browse the repository at this point in the history

  4. Adjust test to new style

    @mre
    mre committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    34858f8 View commit details
    Browse the repository at this point in the history

  5. fmt

    @mre
    mre committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    e20228c View commit details
    Browse the repository at this point in the history

  6. clippy

    @mre
    mre committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    ea371dd View commit details
    Browse the repository at this point in the history

  7. wording

    @mre
    mre committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    d7f347a View commit details
    Browse the repository at this point in the history

  8. fmt

    @mre
    mre committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    e1930cf View commit details
    Browse the repository at this point in the history

  9. clippy

    @mre
    mre committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    1f36ab1 View commit details
    Browse the repository at this point in the history

  10. map_err -> context

    @mre
    mre committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    1acdd24 View commit details
    Browse the repository at this point in the history

  11. Replace FuturesUnordered with ParThenUnordered

    @mre
    mre committed Nov 30, 2021
    Configuration menu
    Copy the full SHA
    ef7d760 View commit details
    Browse the repository at this point in the history

Commits on Dec 1, 2021

  1. Refactoring and add benchmark

    @mre
    mre committed Dec 1, 2021
    Configuration menu
    Copy the full SHA
    69daa59 View commit details
    Browse the repository at this point in the history

Commits on Dec 4, 2021

  1. work on extractor

    @mre
    mre committed Dec 4, 2021
    Configuration menu
    Copy the full SHA
    8a03877 View commit details
    Browse the repository at this point in the history

  2. wip

    @mre
    mre committed Dec 4, 2021
    Configuration menu
    Copy the full SHA
    77750a2 View commit details
    Browse the repository at this point in the history

Commits on Dec 5, 2021

  1. truncat input

    @mre
    mre committed Dec 5, 2021
    Configuration menu
    Copy the full SHA
    3f6408d View commit details
    Browse the repository at this point in the history

  2. truncate

    @mre
    mre committed Dec 5, 2021
    Configuration menu
    Copy the full SHA
    f14ec27 View commit details
    Browse the repository at this point in the history

  3. Use ellipse

    @mre
    mre committed Dec 5, 2021
    Configuration menu
    Copy the full SHA
    f457594 View commit details
    Browse the repository at this point in the history

  4. Tag extraction

    @mre
    mre committed Dec 5, 2021
    Configuration menu
    Copy the full SHA
    4c26508 View commit details
    Browse the repository at this point in the history

  5. kind -> attribute 

    and cleanup
    @mre
    mre committed Dec 5, 2021
    Configuration menu
    Copy the full SHA
    8dfa804 View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2021

  1. wording

    @mre
    mre committed Dec 6, 2021
    Configuration menu
    Copy the full SHA
    9e5c326 View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2021

  1. wip

    @mre
    mre committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    cdfc0f4 View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2021

  1. Move benches to separate crate

    @mre
    mre committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    f6c7eae View commit details
    Browse the repository at this point in the history

  2. docs

    @mre
    mre committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    4e2cf5e View commit details
    Browse the repository at this point in the history

  3. usage notes

    @mre
    mre committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    5c9dac6 View commit details
    Browse the repository at this point in the history

  4. Add element

    @mre
    mre committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    d2f9f08 View commit details
    Browse the repository at this point in the history

  5. Complete link element/attribute combinations

    @mre
    mre committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    5f62bd5 View commit details
    Browse the repository at this point in the history

  6. cleanup

    @mre
    mre committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    9c1e48a View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2021

  1. Remove rayon and stdout lock

    @mre
    mre committed Dec 9, 2021
    Configuration menu
    Copy the full SHA
    9e2a500 View commit details
    Browse the repository at this point in the history

Commits on Dec 10, 2021

  1. Revert "Remove rayon and stdout lock" 

    This reverts commit 9e2a500.
    @mre
    mre committed Dec 10, 2021
    Configuration menu
    Copy the full SHA
    8e2c676 View commit details
    Browse the repository at this point in the history

  2. Revert "Revert "Remove rayon and stdout lock"" 

    This reverts commit 8e2c676.
    @mre
    mre committed Dec 10, 2021
    Configuration menu
    Copy the full SHA
    0502132 View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2021

  1. cleanup

    @mre
    mre committed Dec 12, 2021
    Configuration menu
    Copy the full SHA
    ca727f9 View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2021

  1. refactor

    @mre
    mre committed Dec 14, 2021
    Configuration menu
    Copy the full SHA
    251f1f4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of github.com:lycheeverse/lychee into extractor

    @mre
    mre committed Dec 14, 2021
    Configuration menu
    Copy the full SHA
    f42686a View commit details
    Browse the repository at this point in the history

  3. cleanup

    @mre
    mre committed Dec 14, 2021
    Configuration menu
    Copy the full SHA
    20bb6a1 View commit details
    Browse the repository at this point in the history

  4. fmt

    @mre
    mre committed Dec 14, 2021
    Configuration menu
    Copy the full SHA
    5663e3d View commit details
    Browse the repository at this point in the history

  5. clippy

    @mre
    mre committed Dec 14, 2021
    Configuration menu
    Copy the full SHA
    e7aa681 View commit details
    Browse the repository at this point in the history

  6. remove version pin

    @mre
    mre committed Dec 14, 2021
    Configuration menu
    Copy the full SHA
    ed6a372 View commit details
    Browse the repository at this point in the history

  7. cleanup

    @mre
    mre committed Dec 14, 2021
    Configuration menu
    Copy the full SHA
    713c41e View commit details
    Browse the repository at this point in the history

  8. refactor

    @mre
    mre committed Dec 14, 2021
    Configuration menu
    Copy the full SHA
    6855562 View commit details
    Browse the repository at this point in the history

Commits on Dec 15, 2021

  1. wip

    @mre
    mre committed Dec 15, 2021
    Configuration menu
    Copy the full SHA
    e5c7e42 View commit details
    Browse the repository at this point in the history

  2. Add back tests

    @mre
    mre committed Dec 15, 2021
    Configuration menu
    Copy the full SHA
    a4dd086 View commit details
    Browse the repository at this point in the history

  3. refactor

    @mre
    mre committed Dec 15, 2021
    Configuration menu
    Copy the full SHA
    a94941d View commit details
    Browse the repository at this point in the history

  4. cleanup

    @mre
    mre committed Dec 15, 2021
    Configuration menu
    Copy the full SHA
    10f5fe3 View commit details
    Browse the repository at this point in the history

  5. cleanup

    @mre
    mre committed Dec 15, 2021
    Configuration menu
    Copy the full SHA
    b510f0f View commit details
    Browse the repository at this point in the history

  6. fmt

    @mre
    mre committed Dec 15, 2021
    Configuration menu
    Copy the full SHA
    0e0fc65 View commit details
    Browse the repository at this point in the history

  7. cleanup

    @mre
    mre committed Dec 15, 2021
    Configuration menu
    Copy the full SHA
    0669bc4 View commit details
    Browse the repository at this point in the history