New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a curation tool #12

Open
booyaa opened this Issue Apr 19, 2018 · 23 comments

Comments

@booyaa
Collaborator

booyaa commented Apr 19, 2018

Specification

A tool is required to extract links from a specific tracking issue for a campaign.

An example of these tracking issue can be see on #6.

The tool should be able to extract the links, de-dupe and save as an RSS file (XML). At a minimum it should extract the blog post page title, blog post link, publication date.

We expect the tool to be run repeatedly, if the RSS file already exists it should append new items to the end of the RSS file.

Additional information

Bonus if written in Rust, but any language that can be added to an integration service like travis-ci would be considered.

Mentor: @skade


Mentoring can be provided, just ask.
Please add comment if you are claiming this issue so we can assign it to you.

@booyaa booyaa added help wanted and removed help wanted medium labels May 4, 2018

@booyaa booyaa added this to the MozSprint2018 milestone May 10, 2018

@skade

This comment has been minimized.

Collaborator

skade commented May 10, 2018

I would be willing to mentor this. This is a good first issue for someone wanting to write a whole (small) project in Rust.

@fourplusone

This comment has been minimized.

fourplusone commented May 10, 2018

Would be very interested in writing this tool 😃 in rust

@o0Ignition0o

This comment has been minimized.

o0Ignition0o commented May 10, 2018

This sounds like lots of fun, could I give it a try ?

I'd probably pick an http client such as Actix's or Hyper's, to (async) crawl the page and extract links into a BlogPost struct, then use serde to write / append the xml.

I'll have a look at the RSS spec, could I try to work on this ? :)

@o0Ignition0o

This comment has been minimized.

o0Ignition0o commented May 10, 2018

Oh sorry @fourplusone, please go ahead if you want to :)

@skade

This comment has been minimized.

Collaborator

skade commented May 10, 2018

I would actually recommend just to use reqwest. 👍

Can you please join our gitter channel? That might be easiest.

If you have any questions on the specification, please post them here, so that everyone sees them.

@fourplusone

This comment has been minimized.

fourplusone commented May 10, 2018

I just joined. Should this tool go into a separate repo or should it be part of this one?

@skade

This comment has been minimized.

Collaborator

skade commented May 10, 2018

@fourplusone separate repos is probably the easiest

@fourplusone

This comment has been minimized.

fourplusone commented May 10, 2018

My WIP implementation for this can be found here: https://github.com/fourplusone/curate-issue

@booyaa booyaa moved this from To do to In progress in Mozilla Global Sprint 2018 May 11, 2018

@booyaa booyaa added work-in-progress and removed help wanted labels May 11, 2018

@fourplusone

This comment has been minimized.

fourplusone commented May 12, 2018

Here is a status update of the curate-issue tool.

  • Extracts links from Github Issues + Comments
  • Is able to extend existing RSS Feeds
  • Detects duplicates
  • Extracts Post Date & Title from (most) blog posts
  • Compiles without warnings
  • Has a few test cases
  • Documentation of the code & what it does
  • Moving out more stuff from main.rs
  • Unit Tests which do not rely on Github / Example blog posts
  • Some sort of caching to avoid visiting every page being linked

If you are missing any point, please let me know

@o0Ignition0o

This comment has been minimized.

o0Ignition0o commented May 12, 2018

Awesome !
If you need help or would like me to review it, please let me know :)

@fourplusone

This comment has been minimized.

fourplusone commented May 12, 2018

@o0Ignition0o I'd be glad if you would review some code or improve it

Thanks for your help 👍

@fourplusone

This comment has been minimized.

fourplusone commented May 19, 2018

I think this is ready for testing. Can someone create a repo on /rust-community where the code will find its new home?

@fourplusone

This comment has been minimized.

fourplusone commented May 19, 2018

If you need any help in adding this tool to a .travis.yml file, just let me know

@booyaa

This comment has been minimized.

Collaborator

booyaa commented May 22, 2018

I'll do some testing, will also raise this with the community team about transferring the repo. as this has implications regarding on-going maintenance.

Thanks again for your work!

@booyaa

This comment has been minimized.

Collaborator

booyaa commented May 23, 2018

As discussed with @adityac8 on the irc yesterday, who will be testing the tool and providing feedback to @fourplusone. cc @wezm

17:47 <@booyaa> adityac8: do you want to try the curation tool against the posts you've collected for rustreach?
17:48 < adityac8> Sure. I would love to give that a try.
17:48 <@booyaa> we should raise an issue with readrust's author @wezm just to let him know we're going to do this. he might be able to make our curated posts stand out or create
                a category like "content-o-tron"?
@wezm

This comment has been minimized.

wezm commented May 23, 2018

Might be a bit late given the state of the tool but some of the work I did for Read Rust might be relevant. Specifically the add-url tool and feed finder crate

I’ll give the tool a look when I have a moment and work out a good way to surface posts that are part of a campaign.

@fourplusone

This comment has been minimized.

fourplusone commented May 24, 2018

The feed finder crate looks very useful. I think i will integrate this in an upcoming release.

@booyaa

This comment has been minimized.

Collaborator

booyaa commented Jun 20, 2018

@fourplusone Discuss this with the whole community team, we're happy for you to transfer the ownership of the repo if you still want to do it. Just let us know when it's been done. Thanks!

@adityac8

This comment has been minimized.

Member

adityac8 commented Jul 6, 2018

@badboy I think so we should be transferring this one to rust-community as well 😄
cc @fourplusone @booyaa

@fourplusone

This comment has been minimized.

fourplusone commented Jul 18, 2018

👋 @badboy is there anything you need from me in order to transfer the repo?

@badboy

This comment has been minimized.

Member

badboy commented Jul 18, 2018

@fourplusone Simply transfer the repository to me and I will transfer it to the organization.

@badboy

This comment has been minimized.

Member

badboy commented Jul 18, 2018

@badboy

This comment has been minimized.

Member

badboy commented Jul 18, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment