Extract all unique links from a MediaWiki markup

[yandexx]

Fixes #245.

• I went for deduplicated Vectors instead of HashMaps, but let me know if HashMap is a better choice after all.

[budziq]

Nice work! just few suggestions below:

• please update PR description with fixes #254 to autoclose the issue

[budziq]

Nice work! just few suggestions below:

• please update PR description with fixes #254 to autoclose the issue

[budziq]

I would try to minimize the info regarding the actual mediawiki syntax (ideally removing them all together).
I would just add external link to wikipedia site describing the link syntax.

[budziq]

most likely you can get rid of resp variable by creating the request and reading from it in one go

[budziq]

using map then collect into a set would be preferable

[budziq]

using set would be preferable instead of sorting and deduping

[budziq]

single regex with named capture groups would suffice. collecting results into a single HashSet

[yandexx]

That was the plan but I've tried @\[\[(.+?)(\|?.*?)\]\] and the first group is too lazy and captures only the first character. I'll research further I guess.

[budziq]

most likely you'll have to glue the two regexes into one with |

[yandexx]

But then I can't use a single named group, right? Using it twice panics, as expected.

[yandexx]

Never mind I think I found a solution that works :)

[budziq]

Good progress! Some final stylistic touches left.

also please squash the commits into one.

[budziq]

Good progress! Some final stylistic touches left.

also please squash the commits into one.

[budziq]

please consider extracting the link extraction function with regex compiled under lazy_static! similarly as in the case of soon to be merged PR https://github.com/brson/rust-cookbook/pull/247/files

[yandexx]

lazy_static! doesn't work with the ? operator so I'll have to unwrap? Let me know if there's an alternative.

[budziq]

unwrapping the regex compilation is ok this is an invariant of your code (regex is well written) and not something should try to recover from

[budziq]

I would rewrite it to "MediaWiki link syntax"

[budziq]

i would suggest to have a single regex concatenated with |
to make the regex readable please use the multiline (whitespace insensitive) regex definition like in https://brson.github.io/rust-cookbook/basics.html#run-an-external-command-and-process-stdout (you can split the regex into smaller parts each with comment starting with #)

having different capture groups would actually be beneficial as you will be able to match on them separately and implement the case insensitive logic (to_lower) in one arm.

[yandexx]

Doesn't seem to be possible to use multiline since I use raw string literals.

[yandexx]

Also this is what I have so far and I really don't like the if/else there. My iterator-fu may be lacking :/

    lazy_static! {
        static ref WIKI_REGEX: Regex = 
            Regex::new(r#"\[\[(?P<internal>[^\[\]|]*)[^\[\]]*\]\]|(url=|URL\||\[)(?P<external>http.*?)[ \|}]"#)
            .unwrap();
    }

    let links: HashSet<String> = WIKI_REGEX
        .captures_iter(content.as_str())
        .map(|c| {
            if c.name("internal").is_some() {
                c["internal"].to_lowercase()
            } else {
                // c.name("external").is_some()
                c["external"].to_string()
            }
        })
        .collect();

[budziq]

Doesn't seem to be possible to use multiline since I use raw string literals.

please look at the example from docs

let re = Regex::new(r"(?x)
  (?P<y>\d{4}) # the year
  -
  (?P<m>\d{2}) # the month
  -
  (?P<d>\d{2}) # the day
").unwrap();

it also uses a raw literal. actually the (?x) is critical here.

in regard to the map you have posted I would use a match instead returning Cow

[yandexx]

Thanks, (?x) has helped!

in regard to the map you have posted I would use a match instead returning Cow

Hmm but what would I match on? What to return in case of no match? which shouldn't happen but this case still has to return something?

[budziq]

sorry I meant match returning Cow<str> (damn autocorrect on mobile)
you would write something along the lines of:

match (c["internal"], c["external"]) {
        (Some(val), None) => Cow::from(val.to_lowercase()),
        (None, Some(val)) => Cow::from(val),
        _ => unreachable!(),
    }

if tje unreachable would be triggered then there is a fundamental error with the regex definition so we want to fail noisily and fix our code and not try to recocer

[yandexx]

Ooh that's clever! Thanks, I'm learning a lot here!

[budziq]

i would iterate over the links or use the pretty print syntax

[budziq]

annotating the variable type or the function return type is slightly cleaner.

[brson]

I rebased and merged. Thanks @yandexx