Yet another API attempt

[githubshrek]

Author: Hugh
Tatoeba: https://tatoeba.org/eng/user/profile/tatoebashrek

Introduction

Problem definition

I would like to build an API for Tatoeba to allow easy and direct use by other websites.

I am the owner of the language-learning website https://lingopolo.org/ which is a sort of cross between Anki (flashcard tool), Duolingo (language-learning tool) and Tatoeba. It builds language lessons automatically based on the added sentence content.

Recently we have started to use content from Tatoeba, e.g. see the Tatoeba acknowledgement at the bottom of these pages:

• https://lingopolo.org/thai/word/why-didnt-you-call-me-last-week
• https://lingopolo.org/thai/word/i-ran-across-my-old-teacher-on-the-street
• https://lingopolo.org/dutch/word/tom-hasnt-eaten-enough
• https://lingopolo.org/dutch/word/most-of-my-friends-are-still-single

What we would like to do is automatically add sentences from Tatoeba, directly and programatically from the Lingopolo website. The Lingopolo website would regularly, via cron, make requests to Tatoeba of the type:

• give me the ids of 100 sentences in NL with an EN translation and audio
• for each of the ids, give me the full content of that particular sentence, such as:
  • id
  • EN text
  • NL text
  • copyright information
  • user
  • audio

Proposed solution summary

This describes part of a possible Tatoeba API, which would allow users to directly via API (on a URL):
query the ids of sentences matching a particular tag
query an individual sentence

I see that there have been some attempts at building an API before:

• in 2014, https://github.com/Akirato/tatoeba-api
• in 2015, https://github.com/Tatoeba/tatoeba-api
• in 2020, PoC of a REST API #2341

Lingopolo

Lingopolo consists of 2 developers, myself and Alex. It is based on Drupal (which means it is written in PHP like Tatoeba). We would anticipate that we would do most of the development of this API.

Useful links on API structure

There are many external APIs which it would be worth studying to get the best structure, but here are a few recent links I was looking at.

Wikidata API:

• https://www.wikidata.org/wiki/Wikidata:Data_access

See how Wikidata offers two formats, one which shows HTML and one which shows Json, by adding the "format=json" parameter:

• https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q9128
• https://www.wikidata.org/w/api.php?action=wbgetentities&ids=Q9128&format=json

In the description below I use 2 different formats inconsistently for the API:

• /api.php?action=show_sentences_with_tag=7770&offset=0&limit=100
• GET /sentence/1111401

Get the list of sentences which match a given tag

HTML

https://tatoeba.org/eng/tags/show_sentences_with_tag/7770

Example API call

/api.php?action=show_sentences_with_tag=7770&offset=0&limit=100

Pagination based on https://nordicapis.com/everything-you-need-to-know-about-api-pagination/

Example Response

100 rows in the following format:

{
    "id": 6549421,
    "id": 6627714,
    "id": 6996983,
    "id": 2722194,
    …
}

Get a sentence

This call returns the detail of a single sentence.

Example: "Tom asked me who walked home with me last night."

https://tatoeba.org/eng/sentences/show/6432721

{
    "id": 6432721,
    "text": "Tom asked me who walked home with me last night.",
    "language": "en",
    "derivedFrom": "",
    "belongsTo": "CK",
    "tags": [],
    "textLicence": "CC BY 2.0 FR",
    "audioLicence": "CC BY 2.0 FR",
    "audio": "https://audio.tatoeba.org/sentences/eng/6432721.mp3",
    "translations": [
        {
            "id": 9672097,
	"language": "epo",
	"text": "Tomo demandis al mi, kiu marŝis hejmen kun mi hieraŭ nokte."
        }
     ]
}  

Example "VK is de afkorting van het Verenigd Koninkrijk"

Gets the information for a single sentence such as https://tatoeba.org/eng/sentences/show/1111401

This example demonstrates how the direct and indirect translations would be arranged in two separate arrays.

Example call:
GET /sentence/1111401

Response:

{
    "id": 1111401,
    "lang": "nl",
    "text": "VK is de afkorting van het Verenigd Koninkrijk",
    "textLicence": "CC BY 2.0 FR",
    "audioLicence": "CC BY 2.0 FR",
    "owner": "megamanenm",
    "directTranslations": [
        {
            "id": 1067763,
            "lang": "en",
            "text": "UK is the abbreviation for the United Kingdom"
        }
        {
            "id": 1208155,
            "lang": "esp",
            "text": "UR estas mallongigo por Unuiĝinta Reĝlando."
        }
    ]
    "indirectTranslations": [
        {
            "id": 7527207
 	"lang": "ber",
            "text": "TY d asegzel n Tgelda Yedduklen."
        }
    ]
}

Get user (member)

Get the details of a particular user (member).

HTML

https://tatoeba.org/eng/user/profile/Guybrush88

Example API call

/api.php?action=get_user&user=Guybrush88

Example response

{
    "user": "Guybrush88",
    "country": "Italy",
    "birthday": "1988-12-27",
    "homepage": "",
    "commentsPosted": 6717,
    "sentencesOwned": 687581,
    "audioRecordings": 0,
    "sentencesFavorited": 45,
    "contributions": 2243651,
    "languages": [
        {
        "language": "it",
        "level": 5,
        "details": "Native speaker"
        }
        {
        "language": "en",
        "level": 4,
        "details": "C1 level of Cambridge, C2 of my University"
        }
        {
        "language": "fr",
        "level": 4,
        "details": "C1 level of my University"
        }
        {
        "language": "egl",
        "level": 4,
        "details": "The dialect from my province"
        }
}

Get the list of supported languages

This is not seen as currently required, but is included as one of the many other API endpoints which might eventually be built.

returns something like:

{
     "languages": ["en", "fr", "nl", ....]
}

[jiru]

Thank you so much for sharing your use case with us. We are getting people asking for an API every now and then.

The contents of the json returned by the API look good. Except perhaps the "licence" key. I think we need to make it clearer that there is a license for the text and a license for the audio, by including both of them upfront in the json.

The Lingopolo website would regularly, via cron, make requests to Tatoeba of the type:

• give me the ids of 100 sentences in NL with an EN translation and audio

Could you elaborate on this? What do you do with the sentences, i.e. what’s the complete workflow? Do you want to get new ids at random, or are you trying to get new sentence pairs as they are being added on Tatoeba? When getting the results, how to make sure that you haven’t got some of them already from a previous call?

Additional question: what would be the best way to handle the removal, the edition or the unlink of sentences you previously downloaded using the API?

[githubshrek]

Thanks for the feedback. I didn't realise there were 2 licences.

Licences

I have to say, it's not 100% clear what the first "License" is referring to:

[As an aside, I would suggest to make these 2 headings say "Text" and "Audio" for consistency, and probably add the field label "Licence:" on the Text section, like on the Audio].

Guesswork and reading https://en.wiki.tatoeba.org/articles/show/faq makes me think that the following would describe things correctly:
"textLicence": "CC BY 2.0 FR",
"audioLicence": "CC BY 2.0 FR",

I have changed the original post to use this new format.

Workflow

I have not yet worked out all the details of the Lingopolo end, but I know that if I have 2 API calls available which do:

• give me the sentence data for a given id
• give me a list of Tatoeba sentence ids which match whatever search criteria I wish (basically the Advance Search https://tatoeba.org/eng/sentences/advanced_search via API), e.g.:
  • tag
  • source language
  • target language
  • has audio
  • is approved
  • etc.

then with these 2 API calls I can do everything I would ever want from the Lingopolo end.

Automatic input of all the relevant Tatoeba fields for a single sentence

Let me unpack now vs. future for a single sentence.

Currently, this is what a Lingopolo editor has to do, to go from a sentence in Tatoeba to having that same sentence correctly in Lingopolo:

• (firstly find a suitable sentence)
• create a new Lingopolo entry
• copy the EN text into the Lingopolo entry, into the EN field
• copy the non-EN text into the non-EN field
• copy the Tatoeba licence into the EN text licence field
• (copy the Tatoeba licence into the non-EN text licence field) - this step is currently ignored, and only the EN licence is filled, but this is strictly incorrect, since Lingopolo is using the Tatoeba translation, so that field should also be filled
• download the audio (if exists) from Tatoeba entry
• upload the audio file into the Lingopolo entry
• fill in the audio licence field

The first step would be automating all of this:

• editor on Lingopolo adds the Tatoeba URL (or the sentence id)
• editor clicks on "Get Tatoeba data"

and the API would automatically extract all of the data from Tatoeba, and Lingopolo would automatically fill in all of the fields correctly with the correct data.

Get multiple ids

The next step would be to do this for a bunch of ids, collected from a search, e.g. 100 which match a given criteria selected by the editor.

Get via cron

The final step would be to get this all done via cron.

Answers to your questions

Q. Do you want to get new ids at random, or are you trying to get new sentence pairs as they are being added on Tatoeba?
A. Random is probably fine to begin with. The only thing I have noticed (to my disappointment) is that the quality of some of the translations done by non-native speakers, which my non-native English editors copied and used, not realising that the English was wrong and I had to completely delete. In future I will definitely restrict selection to subsets like:

• "List 907" tag
• "is approved"
• pre-checked lists like http://www.manythings.org/bilingual/

Q. When getting the results, how to make sure that you haven’t got some of them already from a previous call?
There are 2 ways I would do this:

• check that the EN and non-EN are unique
• check the source field to see if it is a Tatoeba link and compare the sentence number

Q. Additional question: what would be the best way to handle the removal, the edition or the unlink of sentences you previously downloaded using the API?
A. I think that's an issue that's 5 years away at least. I suppose you mean if Tatoeba deletes or removes a sentence. In that case, I would probably have some cleanup job that checks the Tatoeba links are still valid

[jiru]

Thank you very much! I now have a good overview of your use case. Search + sentence retrieval is probably the most wanted API feature so it is definitely a good start. I wonder if we want to expose all the advanced search criteria because some criterion like "owner" doesn’t make so much sense outside of Tatoeba. As a course editor, I imagine that you would need to go back and forth between Tatoeba and your own API-connected interface, just to check what owner, tag or list to filter with. This makes me wonder if this approach is such a good way to go for your use case after all. This is just an idea, but since we already provide searching at Tatoeba, what if the course editor would search on tatoeba.org instead, add all the needed sentences to a list (a list is a feature of Tatoeba), and then import that list into the course using a single API call?

By the way your story reminds me that the author of LibreLingo who contacted us recently with what looked like a similar use case.

[mramosch]

I would really like to see e.g. an endpoint that returns a

• [sentence language] to [sentence-owner’s skill level in that language] correlation

i.e. submitting a sentence ID and getting back a language skill level (0, 1, 2, 3, 4, 5 or unspecified)

Or at least a [sentence ID] belongs to [User ID] correlation...

I don’t wanna hit the server with a complete page request for every direct and indirect translation of a sentence and parse all the responses for the usernames of the owner of the translations just in order to circumvent the download of several hundred gigaBytes of download files...

[githubshrek]

I understand where you're coming from. I too am not going to be interested in pulling down Italian sentences owned by a Japanese, or English sentences owned by a Korean etc. so making sure that the owner is a native speaker of the language is clearly an important criteria.

Nonetheless, for a first step, I think it's important to try and keep the API very simple, allowing you to fetch the details of specific entities, in particular a sentence, but it certainly makes sense to also be able to query a particular user (member). I have added to the original description above a "Get user (member)" API call, which would give all the details about a particular user. Since the sentence has a "belongsTo" key then you would be able to go from sentence to get all the details about the sentence owner.

[githubshrek]

OK, I think I've written the draft version of the one part of the requirement, which was:

for each of the ids, give me the full content of that particular sentence, such as:

• id
• EN text
• NL text
• copyright information
• user
• audio

The way to use will be by putting /json/ in the URL instead of /show/ for the sentence, e.g. for sentence 3

• http://localhost:8080/eng/sentences/json/3 show the Json view
• http://localhost:8080/eng/sentences/show/3 show the normal view

Displayed would be the sentence with all translations, e.g.

It seems I only needed to make a few small changes in 2 places:
(1) add 'json' as a URL option for the sentences in auth_actions.php

(2) Write a little json() method to simply dump out the Sentence object:

If this looks OK to folk here, I'll look at how to do the pull request...

[jiru]

If this looks OK to folk here, I'll look at how to do the pull request...

I would be happy to review and merge your code if you create a PR. However, it is not in a state that we can make it the "official" API yet. It would be more like an "unofficial" API only for "insiders" until we can come up with something better. I have been thinking about what’s missing before we can reach this "public" state. First of all, I think the URLs are not good:

The way to use will be by putting /json/ in the URL instead of /show/ for the sentence, e.g. for sentence 3

* http://localhost:8080/eng/sentences/json/3 show the Json view

* http://localhost:8080/eng/sentences/show/3 show the normal view

It looks like you tried to make your API match the website URLs, but I don’t think it’s a good design. Website URLs are not stable and we are free to change them as long as it looks good in everyone’s browser. The URLs of the API, in contrast, should be stable. So better move them to an entire different name space such as a /api prefix (following your naming, that would be https://tatoeba.org/api/sentences/json/3). However, api happens to be a valid language code for Apiaká, so it could create a conflict or prove otherwise confusing. So maybe https://api.tatoeba.org/sentences/json/3 would be better.

I don’t think it is relevant to include "json" in the URL if all the API uses json, so we can shorten it to https://api.tatoeba.org/sentences/3.

Finally, maybe we should include versioning in the URL in case we want to change the API at some point (I am not sure what are the best practices): https://api.tatoeba.org/v1/sentences/3.

All that should be possible to implement using CakePHP routing. For all these reasons, your API code would be better moved into a different controller, for instance ApiController.php.

And then we need to add unit tests for the API end points. For example, a test should check that the API returns a 404 error if the sentence number does not exist or is invalid.

[githubshrek]

I had not read your comment above when I started to implement the "search" part of the API. For that, there wasn't an easy equivalent to the normal /search URL, so I began with /search_json which I rather hated, so I changed to:

• /api/search
• /api/sentence

I think this URL schema is better.

Regarding versioning, yes, I think it is best practice to include a version number. However, since we currently only have one version, I think it's OK too to say that the current URLs will always give the most recent version of the API. You can imagine a future where we have something like:

• /api/search (which now delivers v2)
• /api/v1/search
• /api/v2/search

Regarding tests, yes, no doubt you're right. I haven't included any yet.

[githubshrek]

For the record, here is sample output from the two API calls:

The sentence detail at /api/sentence/[sentence-id]

The search at
http://localhost:8080/eng/api/search?from=eng&has_audio=&native=&orphans=no&query=&sort=relevance&sort_reverse=&tags=&to=fra&trans_filter=limit&trans_has_audio=&trans_link=&trans_orphan=&trans_to=&trans_unapproved=&trans_user=&unapproved=no&user=

which has the identical format to the normal search, other than /sentences/search replaced by /api/search :

http://localhost:8080/eng/sentences/search?from=eng&has_audio=&native=&orphans=no&query=&sort=relevance&sort_reverse=&tags=&to=fra&trans_filter=limit&trans_has_audio=&trans_link=&trans_orphan=&trans_to=&trans_unapproved=&trans_user=&unapproved=no&user=

and will just return an array of sentence ids:

[jiru]

I am glad you are making progress on this. 🙂

Regarding versioning, yes, I think it is best practice to include a version number. However, since we currently only have one version, I think it's OK too to say that the current URLs will always give the most recent version of the API. You can imagine a future where we have something like:

• /api/search (which now delivers v2)
• /api/v1/search
• /api/v2/search

I get your point but I don’t think this is a good idea. Version updates are here to indicate a breaking change. With such setup, if API users use the versionless endpoint /api/search and we update the API to a new version, it will certainly and suddenly break their client. That’s why I think it is better to enforce specifying the version number and making it visible.

[githubshrek]

OK, I'll change to /api/v1/

Here is some reading on the matter:

• https://www.freecodecamp.org/news/how-to-version-a-rest-api/
• https://www.xmatters.com/blog/devops/blog-four-rest-api-versioning-strategies/
• https://restfulapi.net/versioning/

[githubshrek]

I would like to test the search on a large number of sentences (currently I have created 3 in my imouto version).

https://github.com/Tatoeba/tatoeba2/blob/dev/README.tatovm.md says:
"The default install contains no sentences at all and the search won’t work. You will have to add a few sentences on your own."

That's fine for testing with a few sentences, but is there an easy way to import a set? I don't need millions, but I would like to have at least 100 or so to make sure there are no paging issues (e.g. the Json search array being cut off after 10 results).

[jiru]

Unfortunately, you’re on your own for this. There is an open issue: #2010. I am ashamed to admit that I often just run my code on dev.tatoeba.org since I have access. But if it is just a matter of having more than 10 sentences, it’s probably quicker to just add them manually.