API for semantic query

[vecna]

related to #108 this issue is to realize the API needed to fed the map of https://eu19.tracking.exposed

All the URL use paging

/api/v2/:lang/labels/:paging

and this imply you can specify the amount of elements to be skip and to be returned, the order is always reverse chronological.

[vecna]

first release of /api/v2/en/semantics/1-0

change the 1-0 to get more than one post - zero skip. default is 100-0

[
    {
        "_id": "5cb92cf6ab6f23709015fd9f",
        "confidence": 0.7,
        "label": "Finger",
        "lang": "en",
        "semanticId": "1c81fca7a63f1a09f53ba3f5340aa6e4d9639316",
        "spot": "Finger",
        "title": "Finger",
        "when": "2019-04-19T02:05:42.940Z",
        "wp": "http://en.wikipedia.org/wiki/Finger"
    }
]

[vecna]

first release of /api/v2/en/labels/1-0

same consideration above about paging.

[
    {
        "_id": "5cb92cf6ab6f23709015fda0",
        "l": [
            "Hand",
            "Finger"
        ],
        "lang": "en",
        "semanticId": "1c81fca7a63f1a09f53ba3f5340aa6e4d9639316",
        "textsize": 144,
        "when": "2019-04-19T02:05:42.940Z"
    }
]

we need to glue these with eu19 @lrnzctld and I'll provide all the missing information (for example, the post who matches the labels)

[vecna]

The APIs above are the rawest considering our database format, the three high level APIs we need are:

• return the most recent N enrich (labels + summary) analyzed in a language /api/v2/:lang/enrich/:paging
• return the most recent N frequent labels in the last 48 hours (the 48 is hardcoded) /api/v2/:lang/loud/:paging
• return the most recent N enriched containing a specific label /api/v2/:lang/noogle/:label/:paging

they are named: enrich, loud and noogle APIs. @berli0z, when stable they will need the documentation of:

• enrich object
• the three APIs

[vecna]

Loud API /api/v2/en/loud/2-40

this support paging, and returns object in this format:

[
    {
        "count": 87,
        "label": "Brexit",
        "wp": "http://en.wikipedia.org/wiki/Brexit"
    },
    {
        "count": 87,
        "label": "Artificial intelligence",
        "wp": "http://en.wikipedia.org/wiki/Artificial_intelligence"
    }
]

[vecna]

Enirch API /api/v2/fr/enrich/1-2

[
    {
       "l": [
            "Chicago"
        ],
        "lang": "fr",
        "semanticId": "b82a0c9092010c26160fc8ce30c80f48db43557d",
        "summary": {
            "HAHA": 72,
            "LIKE": 327,
            "LOVE": 57,
            "displaySource": "TSUGI",
            "fblinktype": null,
            "images": {
                "captions": [],
                "count": 3
            },
            "nature": "sponsored",
            "permaLink": "/tsugimag/videos/2075270365925449/",
            "postId": "10155878417800755",
            "publicationTime": "2019-04-07T13:00:01.000Z",
            "semanticId": "b82a0c9092010c26160fc8ce30c80f48db43557d",
            "source": "TSUGI",
            "sourceLink": "https://www.facebook.com/tsugimag/",
            "texts": [
                "Quand t'as un procès à juger mais que l'appel de la house est trop fort... Only in Chicago!",
                " via Defected Records"
            ],
            "textsize": 113,
            "videoautoplay": false
        },
        "textsize": 113,
        "when": "2019-04-10T16:20:00.756Z"
    }
]

[vecna]

@berli0z please note, the summary field in the enrich API has less fields than the normal summary (all the potentially linkable information should be redacted). if make sense, we might even call it in a different way then summary

[vecna]

Noogle API /api/v2/en/noogle/Italy/1-12

Paging works, but the right ordering might not (because I'm not using impressionTime as criteria)

[
    {
        "l": [
            "Italy",
            "Canada",
            "Italy",
            "Canada"
        ],
        "lang": "en",
        "semanticId": "8b1aa326f577bc9c92a6fbc087742ae1c3487950",
        "summary": {
            "displaySource": "Canadaic - Canada immigration experts",
            "fblinktype": null,
            "images": {
                "captions": [],
                "count": 1
            },
            "nature": "sponsored",
            "opengraph": {
                "description": "Registration is open",
                "fblinktype": "external",
                "isValid": true,
                "link": "http://www.canadaic.com/registration-case",
                "siteName": "canadaic.com",
                "title": "Check your eligibility"
            },
            "permaLink": null,
            "postId": "362918481232276",
            "publicationTime": "2019-04-08T08:48:17.000Z",
            "semanticId": "8b1aa326f577bc9c92a6fbc087742ae1c3487950",
            "source": "Canadaic - Canada immigration experts",
            "sourceLink": "https://www.facebook.com/Canadaic-Canada-immigration-experts-243305079860284/",
            "texts": [
                "Born in Italy? Canada issues 50,000 permanent resident cards to Applicants from Italy. Holders can live, Work, Study, Travel and open a business in Canada.",
                "Check your eligibility",
                "Registration is open"
            ],
            "textsize": 199
        },
        "textsize": 199,
        "when": "2019-04-10T08:13:14.852Z"
    }
]

@berli0z this should be the same format of enrich. the enriched format is an anonymized post + semantic analysis.

[vecna]

Based on feedback, there API should be improved. now they are not yet ready to be documented, and are considered alpha stage.

[vecna]

Langinfo API /api/v2/es/langinfo

{
    "contributors": 31,
    "hoursWindow": 48,
    "labelsCount": 1694,
    "language": "es",
    "most": [
        "España",
        "Partido Popular",
        "Centro de Investigaciones Sociológicas",
        "Partido Socialista Obrero Español",
        "Familia",
        "Euro",
        "Estado",
        "Ciudadanos-Partido de la Ciudadanía",
        "Venezuela",
        "Policía"
    ]
}

[vecna]

update on langInfo

the API is cache now, two hours each language:

{
    "computedt": "2019-05-02T17:53:27.728Z",
    "content": {
        "consideredHoursWindow": 48,
        "contributors": 0,
        "labelsCount": 0,
        "language": "es",
        "most": [
            "España",
            "Partido Popular",
            "Centro de Investigaciones Sociológicas",
            "Partido Socialista Obrero Español",
            "Familia",
            "Euro",
            "Estado",
            "Ciudadanos-Partido de la Ciudadanía",
            "Venezuela",
            "Policía",
            "Pablo Iglesias Turrión",
            "Partido político",
            "Escaño (elecciones)"
        ]
    },
    "next": "2019-05-02T19:53:27.728Z"
}