New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move record building logic out of WARCWriter. #63

Open
wants to merge 1 commit into
base: develop
from

Conversation

Projects
None yet
3 participants
@isra17
Copy link

isra17 commented Jan 22, 2019

This make it easier creating record without having an handle to a writer.

I wanted to peek first if there's interest upstream. If so I can take care of updating docs and add a few tests.

I can also keep all the create_ methods in WARCWriter to keep it backward compatible.

It also seems to me a few underscore methods in WARCWriter were made to be overridden such as _create_digester, _make_warc_date, _make_warc_id. I can rework the API to make it backward compatible if needed, although what's the point using _private_method convention if not to inform this is not guaranted to be forward compatible ;)

Let me know if this is something you would merge,
Thanks

Move record building logic out of WARCWriter.
This make it easier creating record without having an handle to a
writer.
@codecov

This comment has been minimized.

Copy link

codecov bot commented Jan 22, 2019

Codecov Report

Merging #63 into master will increase coverage by <.01%.
The diff coverage is 96.15%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master     #63      +/-   ##
=========================================
+ Coverage   98.59%   98.6%   +<.01%     
=========================================
  Files          12      13       +1     
  Lines        1354    1358       +4     
  Branches      218     218              
=========================================
+ Hits         1335    1339       +4     
  Misses          1       1              
  Partials       18      18
Impacted Files Coverage Δ
warcio/capture_http.py 97.76% <100%> (+0.03%) ⬆️
warcio/warcwriter.py 97.8% <100%> (+1.22%) ⬆️
warcio/recordbuilder.py 93.97% <93.97%> (ø)
warcio/utils.py 98.93% <98.38%> (-1.07%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c64c439...4af477b. Read the comment docs.

@ikreymer

This comment has been minimized.

Copy link
Member

ikreymer commented Jan 27, 2019

Hey, thanks for starting this! Yes, this definitely makes sense, and something I've thought about but haven't had time to refactor. It makes sense to split the WARC writing from the creation of records.

@wumpus

This comment has been minimized.

Copy link
Contributor

wumpus commented Jan 27, 2019

You made this PR against master and it should have been submitted against develop. I recently made the same mistake! :-)

@isra17 isra17 changed the base branch from master to develop Jan 28, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment