Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

API for semantic query #128

Closed
vecna opened this issue Apr 29, 2019 · 11 comments

Comments

@vecna
Copy link
Member

commented Apr 29, 2019

related to #108 this issue is to realize the API needed to fed the map of https://eu19.tracking.exposed

All the URL use paging

/api/v2/:lang/labels/:paging

and this imply you can specify the amount of elements to be skip and to be returned, the order is always reverse chronological.

@vecna

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

first release of /api/v2/en/semantics/1-0

change the 1-0 to get more than one post - zero skip. default is 100-0

[
    {
        "_id": "5cb92cf6ab6f23709015fd9f",
        "confidence": 0.7,
        "label": "Finger",
        "lang": "en",
        "semanticId": "1c81fca7a63f1a09f53ba3f5340aa6e4d9639316",
        "spot": "Finger",
        "title": "Finger",
        "when": "2019-04-19T02:05:42.940Z",
        "wp": "http://en.wikipedia.org/wiki/Finger"
    }
]
@vecna

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

first release of /api/v2/en/labels/1-0

same consideration above about paging.

[
    {
        "_id": "5cb92cf6ab6f23709015fda0",
        "l": [
            "Hand",
            "Finger"
        ],
        "lang": "en",
        "semanticId": "1c81fca7a63f1a09f53ba3f5340aa6e4d9639316",
        "textsize": 144,
        "when": "2019-04-19T02:05:42.940Z"
    }
]

we need to glue these with eu19 @lrnzctld and I'll provide all the missing information (for example, the post who matches the labels)

@vecna

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

The APIs above are the rawest considering our database format, the three high level APIs we need are:

  • return the most recent N enrich (labels + summary) analyzed in a language /api/v2/:lang/enrich/:paging
  • return the most recent N frequent labels in the last 48 hours (the 48 is hardcoded) /api/v2/:lang/loud/:paging
  • return the most recent N enriched containing a specific label /api/v2/:lang/noogle/:label/:paging

they are named: enrich, loud and noogle APIs. @berli0z, when stable they will need the documentation of:

  • enrich object
  • the three APIs
@vecna

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

Loud API /api/v2/en/loud/2-40

this support paging, and returns object in this format:

[
    {
        "count": 87,
        "label": "Brexit",
        "wp": "http://en.wikipedia.org/wiki/Brexit"
    },
    {
        "count": 87,
        "label": "Artificial intelligence",
        "wp": "http://en.wikipedia.org/wiki/Artificial_intelligence"
    }
]
vecna added a commit that referenced this issue Apr 29, 2019
@vecna

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

Enirch API /api/v2/fr/enrich/1-2

[
    {
       "l": [
            "Chicago"
        ],
        "lang": "fr",
        "semanticId": "b82a0c9092010c26160fc8ce30c80f48db43557d",
        "summary": {
            "HAHA": 72,
            "LIKE": 327,
            "LOVE": 57,
            "displaySource": "TSUGI",
            "fblinktype": null,
            "images": {
                "captions": [],
                "count": 3
            },
            "nature": "sponsored",
            "permaLink": "/tsugimag/videos/2075270365925449/",
            "postId": "10155878417800755",
            "publicationTime": "2019-04-07T13:00:01.000Z",
            "semanticId": "b82a0c9092010c26160fc8ce30c80f48db43557d",
            "source": "TSUGI",
            "sourceLink": "https://www.facebook.com/tsugimag/",
            "texts": [
                "Quand t'as un procès à juger mais que l'appel de la house est trop fort... Only in Chicago!",
                " via Defected Records"
            ],
            "textsize": 113,
            "videoautoplay": false
        },
        "textsize": 113,
        "when": "2019-04-10T16:20:00.756Z"
    }
]
vecna added a commit that referenced this issue Apr 29, 2019
@vecna

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

@berli0z please note, the summary field in the enrich API has less fields than the normal summary (all the potentially linkable information should be redacted). if make sense, we might even call it in a different way then summary

@vecna

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

Noogle API /api/v2/en/noogle/Italy/1-12

Paging works, but the right ordering might not (because I'm not using impressionTime as criteria)

[
    {
        "l": [
            "Italy",
            "Canada",
            "Italy",
            "Canada"
        ],
        "lang": "en",
        "semanticId": "8b1aa326f577bc9c92a6fbc087742ae1c3487950",
        "summary": {
            "displaySource": "Canadaic - Canada immigration experts",
            "fblinktype": null,
            "images": {
                "captions": [],
                "count": 1
            },
            "nature": "sponsored",
            "opengraph": {
                "description": "Registration is open",
                "fblinktype": "external",
                "isValid": true,
                "link": "http://www.canadaic.com/registration-case",
                "siteName": "canadaic.com",
                "title": "Check your eligibility"
            },
            "permaLink": null,
            "postId": "362918481232276",
            "publicationTime": "2019-04-08T08:48:17.000Z",
            "semanticId": "8b1aa326f577bc9c92a6fbc087742ae1c3487950",
            "source": "Canadaic - Canada immigration experts",
            "sourceLink": "https://www.facebook.com/Canadaic-Canada-immigration-experts-243305079860284/",
            "texts": [
                "Born in Italy? Canada issues 50,000 permanent resident cards to Applicants from Italy. Holders can live, Work, Study, Travel and open a business in Canada.",
                "Check your eligibility",
                "Registration is open"
            ],
            "textsize": 199
        },
        "textsize": 199,
        "when": "2019-04-10T08:13:14.852Z"
    }
]

@berli0z this should be the same format of enrich. the enriched format is an anonymized post + semantic analysis.

vecna added a commit that referenced this issue Apr 29, 2019
@vecna

This comment has been minimized.

Copy link
Member Author

commented Apr 29, 2019

Based on feedback, there API should be improved. now they are not yet ready to be documented, and are considered alpha stage.

@vecna vecna closed this Apr 29, 2019
@vecna

This comment has been minimized.

Copy link
Member Author

commented May 2, 2019

Langinfo API /api/v2/es/langinfo

{
    "contributors": 31,
    "hoursWindow": 48,
    "labelsCount": 1694,
    "language": "es",
    "most": [
        "España",
        "Partido Popular",
        "Centro de Investigaciones Sociológicas",
        "Partido Socialista Obrero Español",
        "Familia",
        "Euro",
        "Estado",
        "Ciudadanos-Partido de la Ciudadanía",
        "Venezuela",
        "Policía"
    ]
}
@vecna

This comment has been minimized.

Copy link
Member Author

commented May 2, 2019

update on langInfo

the API is cache now, two hours each language:

{
    "computedt": "2019-05-02T17:53:27.728Z",
    "content": {
        "consideredHoursWindow": 48,
        "contributors": 0,
        "labelsCount": 0,
        "language": "es",
        "most": [
            "España",
            "Partido Popular",
            "Centro de Investigaciones Sociológicas",
            "Partido Socialista Obrero Español",
            "Familia",
            "Euro",
            "Estado",
            "Ciudadanos-Partido de la Ciudadanía",
            "Venezuela",
            "Policía",
            "Pablo Iglesias Turrión",
            "Partido político",
            "Escaño (elecciones)"
        ]
    },
    "next": "2019-05-02T19:53:27.728Z"
}
vecna added a commit that referenced this issue May 2, 2019
@vecna

This comment has been minimized.

Copy link
Member Author

commented May 6, 2019

added two APIs, necessary for https://facebook.tracking.exposed/rss

they are:

/api/v2/languages

and

/api/v2/keywords/:langCode

joxer pushed a commit that referenced this issue May 23, 2019
vecna added a commit that referenced this issue Jun 16, 2019
vecna added a commit that referenced this issue Jun 16, 2019
vecna added a commit that referenced this issue Jun 16, 2019
vecna added a commit that referenced this issue Jun 16, 2019
vecna added a commit that referenced this issue Jun 16, 2019
vecna added a commit that referenced this issue Jun 16, 2019
vecna added a commit that referenced this issue Jun 16, 2019
vecna added a commit that referenced this issue Jun 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.