ADS Classic is being retired #12
Comments
Well, if someone can write the interface with the official API (that is issue #10), yes |
Well, updating |
That's the free software spirit 😉 I am part of the community as much as you, I found something that I could improve here and I contributed to this package. |
Unfortunately I don't know enough elisp, but I think a lot of the existing code can be reused, and instead of parsing the HTML we parse the output JSON from the API. |
|
OK - I started to experiment a bit on how to approach the transition to the new interface. The emacs-request library seems to be what we should be using. After some experimentation I found the interface quite intuitive. However, accessing the API requires an authentication token, which in turn requires the users of this bibslurp-query-ads to have an ADS account. With (defvar auth-token "<token>")
(setq response (request
"https://api.adsabs.harvard.edu/v1/search/query"
:headers
`(("Authorization" . ,(concat "Bearer " auth-token)))
:params
`(("q" . "<query>") ("fl" . "author, title, bibcode"))
:type "GET"
:parser 'json-read)
)Now Using the export-service of the API we can get the bibtex for the individual entries, from their bibcodes.. I can also look into this... Given that I have ZERO previous experience in list, I feel optimistic. P.S: Please feel free to close #10. |
Is it possible to programmatically generate a token? I'm thinking about something similar to what magit forge does |
|
A while ago I started attacking this problem (is still my intention to work of this before October). What needs to be done is straightforward, and most of the work is gymnastics with lists. I have the feeling that considering the userbase of this package it is not unreasonable to ask for a personal ADS API token. That is, I think that the first step should be to modernize the backend. After that, we could maybe worry about the tokens. |
Let's see where they are going with the API. At the moment I think that is the only option we have.
Excellent news - good luck! If you want help with testing and bug-fixing, then create a fork / testing-branch and then I volunteer. @mkmcc will you accept PRs and maintain the repo, or should it be forked? |
|
I may be able to merge pull requests |
|
I finally have some time to spend on this. I forked the repo and implemented some basic functionalities using the APIs. At the moment, it is possible to search for papers and copy the bibtex entry. It is working okay-ish, and there's certainly more work to be done to make it usable. I expect to continue working over the next weekends and hopefully we will have an acceptable replacement before ADS Classic is gone. |
|
FYI: there's a discussion on implementing a backend for ADS (using the new API) in the biblio.el package here: |
|
An update on this. The code I have is pretty much working, even if internally it requires much polishing. @smaret I am not familiar with I am asking with the following in mind. I love That said, I have to add that it is not difficult to complete a new version of |
I can push to this repository. I definitely don't have the time to write myself the new interface, but I'd be happy to review pull requests |
|
I really like the bibslurp workflow. @Sbozzolo thank you so much for working on it. |
They have a similar purpose: fetch references from a bibliographic database and insert it in BibTeX format into a buffer. The main difference is that You can give it a try with e.g. M-x arxiv-lookup "maret 2014"vs M-x bibslurp-query-ads "maret 2014"With that said, I don't mean that |
|
ADS classic is now officially retired, and the 2015 vintage of bibslurp is now sadly broken. How's the port to the new API going? In the meantime, should probably put a notice up on README. |
|
There is a barebone working implementation (https://github.com/Sbozzolo/bibslurp) that supports unstructured queries. You can search and get the bibtex files. The problem is that the way to query ADS is now fundamentally different. Before, I would simply search for an author's name and the year, and I would immediately find the result I was looking for. Now, it is pretty much impossible to find meaningful results without applying filters. At the moment, I haven't implemented a way to use filters. I care about this package and project, and I renew my commitment. However, I had overestimated the amount of time I have, and I have to confess that this is quite a low priority project compared to others. Now, probably things will change considered the retirement of ADS classic, but I cannot give any guarantee. |
|
@Sbozzolo You need to add |
Done 🙂 |
|
OK - the first error was related to my API key entered wrong - the second is that seq-map-indexed is not part of emacs 25.3 - so in order to ensure compatibility with emacs 25: |
|
Thank you very much, I'll commit that change later today.
Knusper writes:
… OK - the first error was related to my error key - the second is that seq-map-indexed is not part of emacs 25.3 - so in order to ensure compatibility with emacs 25:
```
(require 'seq)
(unless (fboundp 'seq-map-indexed)
(defun seq-map-indexed (function sequence)
(let ((index 0))
(seq-map (lambda (elt)
(prog1
(funcall function elt index)
(setq index (1+ index))))
sequence))))
```
|
|
So with this it works for me - I was only using simple queries, and querying for DOI's ... I guess more complicated use cases (incl. the advanced search) may be adressed later... Thanks so far! |
|
Glad to see some progress there. I actually spent a bit of time playing with the new API and mocked up: (require 'request)
(request "https://api.adsabs.harvard.edu/v1/search/query"
:headers '(("Authorization" . "Bearer x....X"))
:params '(( "q" . "author:\"smith,j\"")
("fl" . "bibcode,author,title,abstract,pubdate")
("fq" . "database:astronomy")
("fq" . "property:refereed")) ;notrefereed article
:parser 'json-read
:success (cl-function
(lambda (&key data &allow-other-keys)
(with-current-buffer (get-buffer-create "*ADS BIBSLURP*")
(erase-buffer)
(insert (pp (assq 'response data)))))))which works pretty well. Seems sensible to grab the abstract and just hide it until the user wants to pop it up. I also investigated the links, which are: So similar but different. And you can link to the main ADS abstract page with: So that's worth having. Once you have a bibcode (request "https://api.adsabs.harvard.edu/v1/export/bibtex"
:type "POST"
:data "{\"bibcode\": [\"THEBIBCODE\"]}"
:headers '(("Authorization" . "Bearer x...X")
("Content-Type" . "application/json"))
:parser 'json-read
:success (cl-function
(lambda (&key data &allow-other-keys)
(with-current-buffer (get-buffer-create "*ADS BIBSLURP*")
(erase-buffer)
(insert (cdr (assq 'export data))))))) |
|
For an interface, I think something like transient would be phenomenal. This is the abstracted transient-pop-up interface Magit uses, if you're familiar. It includes all sorts of useful history of former commands (aka entire searches), individual arguments (e.g. authors), etc. Documentation is a bit... opaque. But if you've ever used Magit you'll understand how powerful and intuitive it is. This would be super useful for narrowing down searches, since you can perform a search, see that it's too broad, pop up BibSlurp again set one more filter & search again, etc. And one other brain dump, while I'm thinking of it. By default the new API only returns 10 results, which is probably very speedy. But you can check the |
This is pretty much my implementation in the aforementioned link.
At the moment, I think that the old interface is good enough.
Yes, I had the same thought. I will implement a pager to do what you describe. |
|
The difference here is that JD uses the callback from requests, so we don't
have to block all emacs with :sync t.
…On Mon, Nov 4, 2019, 21:06 Gabriele Bozzola ***@***.***> wrote:
Glad to see some progress there. I actually spent a bit of time playing
with the new API and mocked up:
(require 'request)
(request "https://api.adsabs.harvard.edu/v1/search/query"
:headers '(("Authorization" . "Bearer x....X"))
:params '(( "q" . "author:\"smith,j\"")
("fl" . "bibcode,author,title,abstract,pubdate")
("fq" . "database:astronomy")
("fq" . "property:refereed")) ;notrefereed article
:parser 'json-read
:success (cl-function
(lambda (&key data &allow-other-keys)
(with-current-buffer (get-buffer-create "*ADS BIBSLURP*")
(erase-buffer)
(insert (pp (assq 'response data)))))))
which works pretty well. Seems sensible to grab the abstract and just hide
it until the user wants to pop it up. I also investigated the links, which
are:
https://ui.adsabs.harvard.edu/link_gateway/THEBIBCODE/PUB_PDFhttps://ui.adsabs.harvard.edu/link_gateway/THEBIBCODE/PUB_HTMLhttps://ui.adsabs.harvard.edu/link_gateway/THEBIBCODE/EPRINT_PDFhttps://ui.adsabs.harvard.edu/link_gateway/THEBIBCODE/EPRINT_HTMLhttps://ui.adsabs.harvard.edu/link_gateway/THEBIBCODE/SIMBADhttps://ui.adsabs.harvard.edu/link_gateway/THEBIBCODE/NED
So similar but different. And you can link to the main ADS abstract page
with:
https://ui.adsabs.harvard.edu/abs/THEBIBCODE
So that's worth having. Once you have a bibcode THEBIBCODE, you can grab
the BibTeX ala:
(request "https://api.adsabs.harvard.edu/v1/export/bibtex"
:type "POST"
:data "{\"bibcode\": [\"THEBIBCODE\"]}"
:headers '(("Authorization" . "Bearer x...X")
("Content-Type" . "application/json"))
:parser 'json-read
:success (cl-function
(lambda (&key data &allow-other-keys)
(with-current-buffer (get-buffer-create "*ADS BIBSLURP*")
(erase-buffer)
(insert (cdr (assq 'export data)))))))
This is pretty much my implementation in the aforementioned link.
For an interface, I think something like transient would be phenomenal.
This is the abstracted transient-pop-up interface Magit uses, if you're
familiar. It includes all sorts of useful history of former commands (aka
entire searches), individual arguments (e.g. authors), etc. Documentation
is a bit... opaque. But if you've ever used Magit you'll understand how
powerful and intuitive it is. This would be super useful for narrowing down
searches, since you can perform a search, see that it's too broad, pop up
BibSlurp again set one more filter & search again, etc.
At the moment, I think that the old interface is good enough.
And one other brain dump, while I'm thinking of it. By default the new API
only returns 10 results, which is probably very speedy. But you can check
the numFound key to see if it's larger than 10, and pass in (+ start
numReturned) to get the next page. So perhaps a key binding to load up the
next (or previous) page in the bibslurp buffer would be useful.
Yes, I had the same thought. I will implement a pager to do what you
describe.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#12?email_source=notifications&email_token=ABD4RFS2S77JRP5NWCQZ2ETQSC2HZA5CNFSM4HJRSVS2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDBESQY#issuecomment-549603651>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABD4RFVNNE6IHYYYIIRKPBLQSC2HZANCNFSM4HJRSVSQ>
.
|
|
Sorry hadn't had a chance to look at it yet.
Right. And the callback can kick off formatting the buffer, etc. BTW, the new API is rather powerful, which might be a good motivation for a more feature-rich interface. For example, this search: (require 'request)
(let ((request--url-unreserved-chars
(push 43 request--url-unreserved-chars))) ;The '+' character needs protecting
(request "https://api.adsabs.harvard.edu/v1/search/query"
:headers '(("Authorization" . "Bearer x...X"))
:params '(( "q" . "abs:quasar year:1990- citation_count:1000-")
("sort" . "citation_count+desc")
("fl" . "bibcode,first_author,title,pubdate,citation_count,read_count")
("fq" . "database:astronomy")
("fq" . "property:refereed")) ;
:parser 'json-read
:success (cl-function
(lambda (&key data &allow-other-keys)
(with-current-buffer (get-buffer-create "*ADS BIBSLURP*")
(erase-buffer)
(insert (pp (assq 'response data))))))
:error (cl-function
(lambda (&rest args &key error-thrown &allow-other-keys)
(message "Got error: %S" error-thrown)))))delivers the top 10 quasar-related papers since 1990 sorted by descending citation count. Note that I found that |
I certainly agree with that, and I picture in my head a nice package with transient, as you suggested. My previous comment was probably to be read as "I don't think this is the priority at the moment". I think the top priority is to restore most of the functionalities and remove legacy code.
Yes, you are right |
That is interesting, I the same holds for |
|
So somehow the @Sbozzolo fork now also stopped working for me - slurping some bibtex from ADS fails with the following: Debugger entered--Lisp error: (wrong-type-argument stringp nil)
string-match("@\\sw+{\\([^,]+\\)," nil)
(progn (string-match "@\\sw+{\\([^,]+\\)," bibtex) (replace-match new-label t t bibtex 1))
(progn (progn (string-match "@\\sw+{\\([^,]+\\)," bibtex) (replace-match new-label t t bibtex 1)))
(if (not (string-equal new-label "")) (progn (progn (string-match "@\\sw+{\\([^,]+\\)," bibtex) (replace-match new-label t t bibtex 1))))
(let ((bibtex (bibslurp/request-bibtex bibcode))) (if (not (string-equal new-label "")) (progn (progn (string-match "@\\sw+{\\([^,]+\\)," bibtex) (replace-match new-label t t bibtex 1)))))
(if (not (equal bibslurp-bibtex-label-format 'author-year)) (bibslurp/request-bibtex bibcode) (let ((bibtex (bibslurp/request-bibtex bibcode))) (if (not (string-equal new-label "")) (progn (progn (string-match "@\\sw+{\\([^,]+\\)," bibtex) (replace-match new-label t t bibtex 1))))))
bibslurp/bibcode-to-bibtex("2002AJ....124..266P" "Peng2002")
(kill-new (bibslurp/bibcode-to-bibtex bibcode (bibslurp/suggest-label authors date)))
(let ((bibcode (get-text-property (point) 'bibcode)) (authors (get-text-property (point) 'authors)) (date (get-text-property (point) 'date))) (kill-new (bibslurp/bibcode-to-bibtex bibcode (bibslurp/suggest-label authors date))) (message "Saved bibtex entry to kill-ring."))
bibslurp-slurp-bibtex()
funcall-interactively(bibslurp-slurp-bibtex)
call-interactively(bibslurp-slurp-bibtex nil nil)
command-execute(bibslurp-slurp-bibtex) |
|
Yes, I noticed that too. I haven't found the time to fix it yet. I think that ADS updated the APIs to accept the payload in only a specific way (I didn't see any news about this). I think I have already pinpointed the issue with |
|
I was digging through this as well, and I also couldn't find any update on
the ADS site regarding the change. Very strange.
…On Thu, May 6, 2021, 14:19 Gabriele Bozzola ***@***.***> wrote:
Yes, I noticed that too. I haven't found the time to fix it yet. I think
that ADS updated the APIs to accept the payload in only a specific way (I
didn't see any news about this). I think I have already pinpointed the
issue with curl and I have to implement the change in elisp using the
request module.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#12 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABD4RFS65WER7JVIDQ6VKHDTMLMSNANCNFSM4HJRSVSQ>
.
|
|
The issue is the following. What we do at the moment is more or less (if I remember correctly) What works is So, we need to change the way we pass the data. It shouldn't be too difficult, but I have to understand how |
|
My use of |
The specific issue is for retrieving the And yes, it would be very nice to with transient. |
|
I fixed it in the latest commit. It should be working now. |
Any hope this awesome package will be updated for the new ADS?
The text was updated successfully, but these errors were encountered: