284 MBox Refresher

[ian-lastname]

… helix config in accordance to new save file structure

I have created the parse_mbox_latest_date and refresh_mbox functions. The latter function deletes the latest year and month mbox file that is currently downloaded (identified by parse_mbox_latest_date), and redownloads that along with any file beyond up until the current year. The naming convention of the downloaded files are also changed to what we have agreed on. Just to note, download_mod_mbox REMAINS UNCHANGED since I'm only using download_mod_mbox_per_month.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 206 lines in your changes are missing coverage. Please review.

Project coverage is 36.42%. Comparing base (2bc8d14) to head (b5be04e).
Report is 3 commits behind head on master.

Files	Patch %	Lines
R/mail.R	0.00%	206 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #295      +/-   ##
==========================================
- Coverage   39.79%   36.42%   -3.37%     
==========================================
  Files          20       20              
  Lines        3091     3495     +404     
==========================================
+ Hits         1230     1273      +43     
- Misses       1861     2222     +361     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[carlosparadis]

Thank you @ian-lastname. I will try to make a pass before our meeting!

[carlosparadis]

@ian-lastname thanks!

[daomcgill]

Hi @carlosparadis,

I've refactored the download_pipermail() function.

Proposed Changes

• Removed archive_url and archive_type parameters to simplify the function interface.
• Added start_year_month and end_year_month parameters for date filtering.
• Removed convert_pipermail_to_mbox(), since download_pipermail() handles file conversion automatically.
• Changed the file naming convention to ''kaiaulu_'YYYYMM.mbox' for consistency.

I've added temporary configuration entries in helix.yml for testing purposes:

conf <- yaml::read_yaml("conf/helix.yml")
mailing_list <- conf[["mailing_list"]][["mod_mbox"]][["pipermail_key"]][["mailing_list"]]
start_year_month <- conf[["mailing_list"]][["mod_mbox"]][["pipermail_key"]][["start_year_month"]]
end_year_month <- conf[["mailing_list"]][["mod_mbox"]][["pipermail_key"]][["end_year_month"]]
save_folder_path <- conf[["mailing_list"]][["mod_mbox"]][["pipermail_key"]][["save_folder_path"]]

And this function call:

download_pipermail(
  mailing_list = mailing_list,
  start_year_month = start_year_month,
  end_year_month = end_year_month,
  save_folder_path = save_folder_path
)

Testing Results

• Retrieves links according to date range
• Downloads and renames files from links
• Unzips and cleans gz files
• Saves files as mbox with new naming convention
• Note: I attempted to download and decompress the files directly without saving the .gz to disk, but could not establish a valid connection. The current implementation still downloads the gzipped file and then removes it after unzipping.

[carlosparadis]

@daomcgill I made an inline comment to reply to your question of changes, since there may be a bit of misunderstanding. Let me know if you can't find it.

Two sanity checks:

• Did you try running parse_mbox() to see if it works on the renamed files yet?
• Does running the refresh function pointed at a empty folder downloads the files up to date?
• Does deleting the most recent mbox file download only the newest file back?
• Does running the refresh without deleting any file also update the more recent file only? (to add more recent day's worth of data to the current file)
• Suppose you ran the current refresh today, Sep 16. Then also supposed the next time you executed again the refresh was Oct 2. Would it overwrite the Sep file, and then download a Oct file?

[carlosparadis]

@daomcgill i've added some inline comments, for sanity sake you can reply to each comment directly in line on these since they are more specific.

[carlosparadis]

See inline comments.

[carlosparadis]

More revisions and a few questions.

[carlosparadis]

@daomcgill one of @rnkazman students is needing this to download mod_mbox for Apache Maven. The Notebook documentation is already there right? I know it is functional from the last demo.

[daomcgill]

@carlosparadis the updates are finalized and awaiting your review. @rnkazman please let me know if your student runs into any issues using it for Apache Mavern.

[carlosparadis]

@daomcgill thanks! It's been on my to-do list on the code side but I was happy to see the refresh was in order from our last group call. You did get the 10 page long e-mail of M3 to have something as backup on the meantime, right?

[daomcgill]

@carlosparadis I received it, thank you. I am working on figuring out where to start, and should have an issue up for it soon.

[carlosparadis]

@daomcgill no worries. Remember, you can always start from a semi empty issue and we can q&a out of it on it.

[carlosparadis]

@daomcgill

A couple things:

• Narrative of the notebook still needs some massage. See the text I added at the beginning as an starting point. Move the config file to the top and brief the user in one go from there instead of showing in parts for the different archives. It should be relatively clear from the name in the new config they are separate.

• New config specification and gets are still missing. I recommend at least use the new config format without the get, so that you can write the narrative with the proper explanation so I can review.

• Many of my commits above is trying to resolve why Actions is failing. Take a look at the commit messages to see what I tried. You will need to revert to OS 13 (i.e. see what I changed on the last commit and undo). From there, you will need to figure out why the gcc compiler is not working on Actions anymore.

This used to work, but it is not the first time Actions stop working. Moving to 14 is a no-go because of another issue with uctags, a separate dependency of Kaiaulu that I would like us to still be able to test.

The good news is that the checks passed on my machine, but we can't have my offline computer being the only way for your group to run Checks and unit tests.