Turning URLs into links

[mfenner]

I want URLs in citations to be links, and I have monkey patched the reference_tag method to allow this:

module Jekyll
  class Scholar
    module Utilities

      alias_method :original_reference_tag, :reference_tag

      def reference_tag(entry)
        return missing_reference unless entry

        # Change URLs into markdown links
        entry["url"] = "[#{entry["url"]}](#{entry["url"]})" if entry["url"]

        entry = entry.convert(*bibtex_filters) unless bibtex_filters.empty?
        reference = CiteProc.process entry.to_citeproc,
          :style => style, :locale => config['locale'], :format => 'html'

        content_tag reference_tagname, reference,
          :id => [prefix, entry.key].compact.join('-')
      end
    end
  end
end

You can see that I change entry["url"] into a markdown link, which is then picked up my the markdown processor. You can see this at work at http://blog.martinfenner.org/about.html. Is this the correct way of doing this? I'm using Pandoc as markdown processor and in the default configuration it doesn't recognize blank URLs.

We could do something similar for entry["doi"], as some styles display the DOI.

[inukshuk]

Absolutely. Jekyll-Scholar started out as very quick plugin — the idea was that it should allow you to re-use the utilities and override selected methods if you need custom behaviour. So your solution is basically what I assumed people would do. Another cool solution I've seen is by @ashinkarov here #23 using a custom CSL style. When we finally finish the citeproc-ruby rewrite it will be much easier to hook into the cite processing and adapt individual components.

As far as adding such customizations to the jekyll-scholar gem I'm not sure if it is really worth it to support this functionality and not better to make it easy for people to override everything to their use case. The big problem here is that I'm probably the least experienced jekyll-scholar user myself.

My rule of thumb is that I'll gladly add anything that is generic enough to be useful to most users as long as it is not difficult to maintain, so if you turn this into a pull request and add a test to make maintenance easier I'll gladly merge it :-)

[ashinkarov]

I think that the problem is that the proposed solution is not generic enough. For example on my page, I put "[url]" text that leads to the paper I am referring to. Assuming your patch would be merged-in, my publication list would be broken.

I believe it is arbitrary hard to hard-code this sort of things to make everyone happy. So as I wrote earlier, the solution that we really need is a dynamic modification of csl files. In essence it's all about representation of the data -- we are not changing the essence of the data.

[mfenner]

I am confused, or maybe I don't understand CSL enough. What I am thinking I propose is that every time a URL is written to the markdown file we are generating, we add standard markdown tags for a link around it. Why should this conflict with other URLs in the CSL? And is there a case for URLs in citations to not be links?

[ashinkarov]

CSL is just a template which is filled with the data from each individual bibitem. For example the content of the title may be taken in pads, or italic might be applied or something else. When it comes to URLs, if not being ignored, then they are passed as clear text, or adding "url:" prefix (it all depends on the style we are using).

Now you are proposing to preprocess url content and instead of clear text, you want to give '<a href="$url">$url</a>' to the CSL file. As a result you will have an effect as you have on your webpage. CSL would just put this text in the overall html.

Now, my problem here would be, that I don't want to see '<a href="$url">$url</a>', but I rather would want to see '<a ref="$url">[url]</a>' like I have on my page: http://ashinkarov.github.io/publications/
The other guy might want to present the url in yet another way on his page.

If you do this processing step at the level of CSL, then you don't have to modify the data that is comming to this CSL. Now, if you hardcode these things, you will need to find a way to parametrise your code, which might or might not cover all the cases.

So, the generic solution would be to specify xml code snippets in your config.yml (or somewhere else), which would add/replace parts of XML tree of the CSL files.

I hope that I explained it well enough. Feel free to ask more.

[inukshuk]

In general I tend to agree with @ashinkarov here that reference data should be kept clean of any sort of markup (html, markdown, latex or otherwise); however, in this case this is completely internal to jekyll-scholar and jekyll-scholar as a blogging tool obviously generates html via markdown. So I think the proposed solution is a pretty neat way of turning the URLs into proper links by default.

Actually, I have thought about this a little and we may even accomplish this as a filter in bibtex-ruby; we could then simply add the name of the filter to the jekyll-scholar configuration to preprocess URLs and DOIs in this way. This way you wouldn't even need to override reference_tag and at the same time it would not interfere with other approaches at all.

@mfenner do you see any problem if the conversion is applied 'permanently' (of course only to the bibliography in memory not on disk) instead of every time in reference tag? Then we could just add a filter to handle the conversion after the bibliography is parsed.

[mfenner]

@ashinkarov your arguments make sense. I am using plain CSL styles without modifications so wasn't considering custom formatting.

@inukshuk a filter in bibtex-ruby would be fine. I can also continue monkey patching jekyll-scholar.

[inukshuk]

@mfenner if you like you could you try something like this. Put a file markdown.rb (file name is not relevant) into your ext directory. There you need to create a class Markdown like this:

module YourNamespaceHere
  class Markdown < BibTeX::Filter
    def apply(value)
      return value unless value.to_s =~ ADD-URL-PATTERN-HERE
      value.to_s.gsub(/ADD_URL_PATTERN_HERE/, '[$1]($1)')
    end
  end
end

And then in your jekyll configuration set:

scholar:
  bibtex_filters:
    - latex
    - markdown

This would turn all values (not only in the URL field) into a markdown link permanently as far as jekyll scholar is concerned. If this works I think we could add this filter to jekyll-scholar by default for you to enable with the bibtex_filters options on demand.

[mfenner]

@inukshuk My _plugins/jekyll_scholar.rb now looks like this:

require 'jekyll/scholar'
require 'uri'

module MarkdownFilter
  class Markdown < BibTeX::Filter
    def apply(value)
      url = value.to_s.slice(URI.regexp(['http','https','ftp']))
      value = url ? "[#{url}](#{url})" : value
    end
  end
end

This works and is now live on my jekyll blog. I have two small issues:

• I'm replacing value with url. I'm not sure whether an URL can be part of a field in Citeproc, e.g. the title.
• This code has problems with old DOIs in funky formats e.g.

    http://dx.doi.org/10.1002/(SICI)1096-9098(199702)64:2<122::AID-JSO6>3.0.CO;2-D 

Auto-linking with Github-flavored markdown is also broken for this URL.

[inukshuk]

Nice! I did not know about URI.regexp

The issue I see is that, in theory, you could have something like this (in the note field for instance):

Available at https://a.tld/b and at http://c.tld/d

That's why I would rather replace the URLs with #gsub. I guess this pertains to the point you raised above as well. I wouldn't know that a CSL processor makes any assumptions as to whether or not a field contains a URL so as far as I know all processors should just pass the markdown link through unscathed.

As for the second issue, perhaps we can add a separate pattern for DOIs; something like /\b(http[^\s]+doi.org\/[^\s]+)/?

[mfenner]

OK, I changed the code to do replace the URL, even if part of the field value:

require 'jekyll/scholar'
require 'uri'

module MarkdownFilter
  class Markdown < BibTeX::Filter
    def apply(value)
      url = value.to_s.slice(URI.regexp(['http','https','ftp']))
      return value unless url
      value.to_s.gsub(/(#{url})/, '[\1](\1)')
    end
  end
end

URI.regexp currently doesn't find some URLs, e.g. those that contain parentheses (because they confuse the regex matching) or really strange things like the URL above. Too bad, but least they are displayed as text. Will look at other patterns for URL matching.

[mfenner]

OK, now I have the bibtex filter working the way I want:

require 'jekyll/scholar'
require 'uri'

module MarkdownFilter
  class Markdown < BibTeX::Filter
    def apply(value)
      value.to_s.gsub(URI.regexp(['http','https','ftp'])) { |c| "[#{$&}](#{$&})" }
    end
  end
end

This code is pretty dense, but I didn't want to regex twice. This only works with valid URLs, but I see that as a feature, not a bug.

The problem of DOIs in funky formats I solved differently as I produced the URL myself: I changed the code to generate a URL from a DOI:

url = Addressable::URI.escape "http://dx.doi.org/#{doi}"

instead of

url = "http://dx.doi.org/#{doi}"

URI.escape is depreciated and CGI.escape does something slightly different, therefore the addressable gem.

[inukshuk]

This looks pretty good!

I think it would be great to add the filter to jekyll-scholar — the only question is whether or not we should enable it by default.

[ashinkarov]

That looks good, assuming that URI.regexp works fine on various links :)
Regarding enabling by default -- can we have a simple flag which would allow to toggle this behaviour. I don't mind if it's turned on by default, if it's documented and I can turn it off easily.

Cheers,
Artem.

[inukshuk]

Yes, the nice thing is that @mfenner turned it into a BibTeX-Ruby filter so we already have the option in place: bibtex_filter — right now it's default is set to [:latex] and we could change it to [:latex, :markdown]. If you don't need it you can simply set it to the current default option again.

[ashinkarov]

Ah, ok, I didn't remember that we have bibtex_filter available via configuration.

[mfenner]

@inukshuk will you add this filter to bibtex-ruby or jekyll-scholar? It would be good to add Cucumber tests to confirm that this filter can handle a variety of links. I would be happy to write the tests, but not before the weekend.

I'm for enabling by default since jekyll-scholar is always using markdown.

[inukshuk]

@mfenner cucumber tests would be great. I was thinking of adding it to jekyll-scholar and I also think it would be a reasonable default — we just need to make sure people know how to switch to the old configuration.

[mfenner]

OK, give me a few days and I will try a pull request that includes cucumber tests. Unless you want to add the filter yourself, then the pull request will only contain the tests.

[inukshuk]

No hurry : ) just let me know if I can do anything to help.

[x12a1f]

I tried this and it just shows the markup in the bibliography list.

I have:

@article{test,
      author = {test},
      title = {test},
      URL = {http://google.com}
}

which shows as [1]test, test, (n.d.). [http://google.com](http://google.com). without converting the url into a clickable link.

Do I need to configure something else?

[inukshuk]

I think we never actually added the filter, so what you need to do is add it to your _plugins/ext.rb file:

require 'jekyll/scholar'
require 'uri'

module MarkdownFilter
  class Markdown < BibTeX::Filter
    def apply(value)
      value.to_s.gsub(URI.regexp(['http','https','ftp'])) { |c| "[#{$&}](#{$&})" }
    end
  end
end

And then enable it in _config.yml:

scholar:
  bibtex_filters:
    - latex
    - markdown

[x12a1f]

yes, that is what I did

[inukshuk]

@mfenner is it OK if I add the plugin? Looks like I should write a test for it anyway : )

[inukshuk]

I think for this to work the way @mfenner intended this needs to happen inside a markdown file, because you need the markdown processor to then convert the link.

Alternatively, you could adjust the filter above to generate an HTML link tag instead. Something like:

value.to_s.gsub(URI.regexp(['http','https','ftp'])) { |c| "<a href=\"#{$&}\">#{$&}</a>" }

Better yet, you could adjust your bibliography template

[mfenner]

@inukshuk fee free to add the plugin.

[x12a1f]

Changing the filter to generate a HTML link tag works.

Thanks.

[benswift]

1. The item is read from your BibTeX file
2. BibTeX filters are applied (e.g. URLs converted to Markdown syntax)
3. BibTeX item is converted to CSL JSON
4. Item is rendered using CSL Style
5. The reference and the BibTeX item are passed to the template
6. The template is further processed by Jekyll (e.g., Markdown turned into HTML)

Hmm, it doesn't seem to be working this way for me. I have modified my .csl file, and can insert the appropriate markdown markup for a standalone link (i.e. <http://example.com>, but it just gets rendered as-is in the output (i.e. it doesn't look like step 6 above is happening.

If you're interested, here's the CSL style, my config file and the resulting output.

[inukshuk]

The angle brackets are inserted into your markdown documents as HTML entities < and >. It would be easier to use [] instead.

[benswift]

Hey, thanks for taking the time to help out.

I tried this in the apa.csl file:

<text variable="DOI" prefix="[link](https://doi.org/" suffix=")"/>

and it renders out like (e.g.) this in the final _site folder:

<li><span id="swiftVisualCodeAnnotations2013">Swift, B., Sorensen, A., Gardner, H. J., &amp; Hosking, J. (2013). Visual Code Annotations for Cyberphysical Programming. In <i>2013 1st International Workshop on Live Programming (LIVE)</i> (pp. 27–30). [link](https://doi.org/10.1109/LIVE.2013.6617345)</span></li>

so it doesn't seem to be getting markdownified.

The "using < and >" thing was based on this example linked in an earlier comment, but as I already mentioned that doesn't work either.

Or maybe I'm just doing it wrong.

[inukshuk]

OK I just tried this using the markdown filter above and I noticed two things: first, if you're using the kramdown Markdown engine, you need to enable parsing inside of HTML blocks (that's disabled by default); and secondly... the output of from the liquid tag does not seem to be run through markdown anymore, like you say. This is probably due to a change in Jekyll?

[benswift]

Yep, I tried parse_block_html: true and tried modified the .csl file to output both markdown and html links, no success either way.

It's not a dealbreaker---people can copy-paste it into their url bar 😉 So I think I'll park it for now. Thanks for the work you did to investigate.

[benswift]

If it's helpful for others who come across this, my blog source is up at https://github.com/benswift/benswift.github.io.

And here's the gory details about all the gem versions from bundle show

  * activesupport (5.2.2)
  * addressable (2.5.2)
  * autoprefixer-rails (9.4.5)
  * bibtex-ruby (4.4.7)
  * bundler (1.17.2)
  * citeproc (1.0.9)
  * citeproc-ruby (1.1.10)
  * colorator (1.1.0)
  * colorize (0.8.1)
  * concurrent-ruby (1.1.4)
  * csl (1.5.0)
  * csl-styles (1.0.1.9)
  * em-websocket (0.5.1)
  * ethon (0.12.0)
  * eventmachine (1.2.7)
  * execjs (2.7.0)
  * extras (0.3.0)
  * fastimage (2.1.5)
  * ffi (1.10.0)
  * forwardable-extended (2.6.0)
  * html-proofer (3.10.0)
  * http_parser.rb (0.6.0)
  * i18n (0.9.5)
  * jekyll (3.8.5)
  * jekyll-assets (3.0.12)
  * jekyll-compose (0.9.0)
  * jekyll-feed (0.11.0)
  * jekyll-paginate-v2 (2.1.0)
  * jekyll-sanity (1.2.0)
  * jekyll-sass-converter (1.5.2)
  * jekyll-scholar (5.14.0)
  * jekyll-seo-tag (2.5.0)
  * jekyll-watch (2.1.2)
  * kramdown (1.17.0)
  * latex-decode (0.3.1)
  * liquid (4.0.1)
  * liquid-tag-parser (1.9.0)
  * listen (3.1.5)
  * mercenary (0.3.6)
  * mini_portile2 (2.4.0)
  * minitest (5.11.3)
  * namae (1.0.1)
  * nokogiri (1.10.1)
  * parallel (1.12.1)
  * pathutil (0.16.2)
  * public_suffix (3.0.3)
  * rack (2.0.6)
  * rb-fsevent (0.10.3)
  * rb-inotify (0.10.0)
  * rouge (3.3.0)
  * ruby_dep (1.5.0)
  * safe_yaml (1.0.4)
  * sass (3.7.3)
  * sass-listen (4.0.0)
  * sprockets (3.7.2)
  * thread_safe (0.3.6)
  * typhoeus (1.3.1)
  * tzinfo (1.2.5)
  * yell (2.0.7)

[grauschnabel]

hi @benswift , at least on your research page the urls ar not clickable for me, so please tell me exactly where to look to solve the problem of this thread. Thanks very much!

[benswift]

I didn’t solve it, sadly. And I tried all the suggestions, so I’m not 100% where to go from here. Sorry.

[danjjl]

Has anybody with Jekyll experience figured out how to get the output of the liquid tag to be processed by the markdown engine?

For the moment {% bibliography --file references %} produces ..... [http://google.com](http://google.com]

-> the output of markdown_links.rb is not processed by the markdown engine.

[rawlins]

Like others the markdown link plugin is not working for me, even though the order of operations documentation implies it should be fine. However, html output is fine, so it wasn't too hard to modify that plugin to generate links that way. The following does what's needed in jekyll 4.0.0 (you can put this as a .rb file in the _plugins folder of your site):

# Contributed by @mfenner as markdown_links.rb
# See https://github.com/inukshuk/jekyll-scholar/issues/30
# modified to generate html instead of markdown by @rawlins

require 'uri'

module Jekyll
  class Scholar
    class HtmlLinks < BibTeX::Filter
      URL_PATTERN = URI.regexp(['http', 'https', 'ftp'])
      def apply(value)
        value.to_s.gsub(URL_PATTERN) {
          "<a href=\"#{$&}\">#{$&}</a>"
        }
      end
    end
  end
end

Does not handle DOIs for the ordering reasons described above, I'm still trying to figure out what would be needed for them for my own purposes.

[rawlins]

Ok, here's a solution for DOIs that works for me, though it has some limitations. First, if you are using a CSL that inserts URL material into the DOI field, you will need to disable that. For example, the one I'm using had a conditional:

          <if variable="DOI">
            <text variable="DOI" prefix="DOI: https://doi.org/" /> -->
          </if>

so I made a copy of this CSL file in my site directory structure, changed the config so it was using this copy, changed this rule to to:

          <if variable="DOI">
            <text variable="DOI" prefix="DOI: " />
          </if>

Second, I installed the following plugin into my site's _plugins folder and added it to the list of bibtex_filters in the config file:

# Convert DOIs to html. This *will not match all DOIs*. The regex comes from
# https://www.crossref.org/blog/dois-and-matching-regular-expressions/
# this may conflict with some CSLs that insert `https://doi.org/` or the like.

module Jekyll
  class Scholar
    class DOILinks < BibTeX::Filter
      DOI_FILTER = Regexp.new("^10\\.\\d{4,9}/[-._;()/:A-Z0-9]+$", Regexp::IGNORECASE)
      def apply(value)
        value.to_s.gsub(DOI_FILTER) {
          "<a href=\"https://doi.org/#{$&}\">#{$&}</a>"
        }
      end
    end
  end
end

As noted, this won't match exotic DOIs (see the link in the code comments above for discussion). For purposes of bibtex value strings, I actually suspect that something much more relaxed like "^10\\.\\d{3,9}/\\S+$" might work just fine, as long as you dealt with html escaping. I can't really think of cases where you'd get false positives for this given that it's not trying to match free text.

[m-pilia]

The approach mentioned by @benswift did not work for me either, but I think I found a way to go around this issue by writing a custom Liquid filter:

# put inside file: _plugin/link_to.rb
module Jekyll::CustomFilter
  def link_to(input, href)
    "<a href=\"#{href}\">#{input}</a>"
  end
end
Liquid::Template.register_filter(Jekyll::CustomFilter)

Then the link_to filter can be used inside the .cls file. For instance, I wanted to create an hyperlink in the title of the citation whenever a url is present, therefore I changed the relevant part of the title macro as follows:

<if variable="URL">
    <text variable="title" quotes="true" prefix="{{ '" suffix="' | "/>
    <text variable="URL" prefix="link_to: '" suffix="' }}"/>
</if>
<else>
    <text variable="title" quotes="true"/>
</else>

It seems to work, but it feels more like a horrible workaround rather than a proper solution.

[inukshuk]

@m-pilia have you tried this in the latest version of jekyll-scholar? I think this will not work anymore, because we just disabled the double parsing of templates.

I think the best option is to use a custom bibliography template; this gives you full access to the entry and to the rendered reference. If you absolutely need the link to be inside the rendered reference, I think the best solution right now is to hack the style, similar to your own example. You can use this code to make it possible to insert HTML snippets into the prefix/suffix attributes (using either '<' or '<' should work). (We should probably add something like this to jekyll-scholar behind an option.)

[m-pilia]

@inukshuk Thanks for the reply! Indeed, I tried and I can confirm that using Liquid filters inside the CSL does not work on the latetst version. Your suggestion works better though.

[drscotthawley]

Has a solution to this ever been made 'standard', or are we still stuck with adding custom hacks? Latest jekyll-scholar, with default (apa) style, still fails on @x12a1f 's simple test case:

@article{test,
      author = {test},
      title = {test},
      URL = {http://google.com}
}

Still produces the bibliography entry that reads...

test. test. [http://google.com](http://google.com)

which is never converted to HTML. (i.e. no links).

...this is even with markdown and latex enabled in config, as per @inukshuk 's instructions:

scholar:
  source: _bibliography
  bibtex_filters:
    - markdown
    - latex

Turning on markdown filter or not doing so, seems to have no effect.

Only solution I can find is to remove the markdown filter and then manually edit the BibTeX to include the HTML:

@article{test,
      author = {test},
      title = {test},
      URL = {<a href="http://google.com">http://google.com</a>}
}

...not ideal.

[quattrinili]

I modified the markdown filter to produce the proper HTML, quattrinili@de8cc5d

I am still testing it to see if that works in general, and probably the right way to do it is to change the name of that filter, but at least it is producing the correct output.

[ashsrao]

I would like to know the steps required to make the title text point the doi. All my bibtex entries have the doi field and I would like to use them.

I updated my custom csl file as follows

<macro name="title">
    <choose>
        <text variable="title"/>
        <!-- with URLs https://github.com/ashinkarov/ashinkarov.github.com/blob/source/plugins/ieee.csl#L180 -->
        <!-- <text variable="DOI" prefix="&lt;a href=&#34;https://dx.doi.org/"/>
             <text variable="title" prefix="&#34;&gt;" suffix="&lt;/a&gt;"/> -->
        <!-- <text variable="title" prefix="[" suffix="]"/>
             <text variable="DOI" prefix="(https://dx.doi.org/" suffix=")"/> -->
    </choose>
  </macro>

And my _config.yml has the latex and markdown filters enabled.

scholar:
 ...
  bibtex_filters:
    - latex
   

I am using the minima theme and kramdown.

I tried the html tags <a href ... , and even markdown tags [title](https://dx.doi.org/doi) but the html pages do not make the titles as links.

Any suggestions would be useful. I am using jekyll 4.2.0 and jekyll-scholar (6.8.0).

[AlasdairGray]

Is there something that needs to be done to get the custom ruby files in _plugins to run?

I've been following the instructions of @m-pilia in this blog. At the moment I see the HTML string rather than it being interpreted by the browser.

doi: <a href="https://doi.org/10.1007/978-3-030-21348-0">10.1007/978-3-030-21348-0</a>

[inukshuk]

All Ruby files in _plugins should get loaded by Jekyll at startup.

Regarding the link, I guess the solution mentioned in the blog is to downgrade to a previous version of jekyll-scholar. In general, I'd try to remove the DOI from the CSL style and use a custom bibliography template to display the rendered reference and then add your own links for URLs, DOIs and similar.

[AlasdairGray]

Thanks, I ended up using the template approach. I hadn't picked up on that in the README. It would be handy to give the set of keys that are available or a pointer to a page that gives them.

[KonstantinBob]

I've been following the instructions of @m-pilia in this blog. At the moment I see the HTML string rather than it being interpreted by the browser.

doi: <a href="https://doi.org/10.1007/978-3-030-21348-0">10.1007/978-3-030-21348-0</a>

I was stuck with this, too. Here is a hack that worked at least for my situation:
In my post layout I replaced

<div class="post-content" itemprop="articleBody">
 {{ content }}
</div>

with

<div class="post-content" itemprop="articleBody">
 {{ content | replace: "&amp;", "&" | replace: "&lt;", "<" | replace: "&gt;", ">" | markdownify }}
</div>

and now the links are rendered correctly.

[amaury-d]

I had the same issue and built this hopefully elegant enough solution (no plugin)

_bibliography/references.bib

@article{test,
  author     = {test},
  title      = {test},
  hidden_url = {http://google.com}
}

_config.yml

scholar:
  bibliography_template: bib

_layouts/bib.html

---
---

{% if entry.hidden_url %}
<a href="{{ entry.hidden_url }}">{{reference}}</a>
{% else %} {{reference}} {% endif %}