Implement a Data Source Interface and add the data from the FOSSASIA and Freifunk Api to loklak through the interface

[mariobehling]

Here is an idea draft how data can be added through the interface.

This issue is related to #205

[Orbiter]

the server part which receives this information is described in loklak/loklak_server#35

[zyzo]

FOSSASIA/Freifunk API data consist of community json files that follow this specification https://github.com/fossasia/api.fossasia.net/blob/master/specs/1.0.1.json.

To be short, this is the complete list of properties :

1. api metadata (api version)
2. community name
3. metacommunity (bigger community that this community belongs to)
4. location (city, country, address, coordinates)
5. contact info (email, facebook, twitter, phone, irc, ..)
6. webpage url
7. timeline (lifetime events something like date of birth or other important milestones)
8. feeds (blog, forum, ics, podcast, wiki, ..)
9. state : the state of the json file (last-changed, open-closed for public), logo, community description

List of all available jsons :
fossasia : https://github.com/fossasia/directory.api.fossasia.net/blob/master/directory.json
freifunk : https://github.com/freifunk/directory.api.freifunk.net/blob/master/directory.json

[zyzo]

To integrate FOSSASIA/Freifunk API data into loklak, a few questions come up :

• what kind of adapter / processor need to be written to dump the data into loklak ?
• where's the part of code should I look at for examples of these kinds of adapter ?

Just to clarify things a bit, API data consist of a lot of information about a community, like names, location, services. From the name and location, a community map was built, and now we want to add some filters to it (by country, by tech area..). Sometimes we want to retrieve data directly from services as well (rss, ics feeds..).

[mariobehling]

@zyzo

what kind of adapter / processor need to be written to dump the data into loklak ?

What is your proposal?

where's the part of code should I look at for examples of these kinds of adapter ?

We are just getting started with it.

Just to clarify things a bit, API data consist of a lot of information about a community, like names, location, services. From the name and location, a community map was built, and now we want to add some filters to it (by country, by tech area..). Sometimes we want to retrieve data directly from services as well (rss, ics feeds..).

How can we pump all this data to loklak? Actually we diverted from the original meaning of "API" a bit and also included more to it. In my opinion - let's focus with the API on the core topic and let other services display the data e.g. the Freifunk blog, Loklak and others.

[mariobehling]

@sudheesh001 Why has this been closed? Where can I test the implementation?

[sudheesh001]

@mariobehling I think this got closed because of the term Fix #206 which was present in the pull request content #259 I've been adding a label Checked in case it works and is actually supposed to be closed.

[zyzo]

@sudheesh001 I think you meant #260 ?

[zyzo]

This went live http://test.loklak.net/dataConnect

[zyzo]

This is what current implementation looks like, with loklak_server new feature loklak/loklak_server#83:

TODO :

• option to modify the datasource directly from web client.
• for geojson source, there is map_type parameter. User should be able to visualize it and modify it as well. This feature requires some more work on the server as the information about map_type is currently not saved.
• when clicking on some item on the table the corresponding information should open. For e.g. when clicking on number of messages the list of messages should open, when clicking on source_type only data source from that source_type should be displayed, etc.

[zyzo]

This feature can be reviewed at http://loklak-client.danghaian.com/dataConnect
(Server : http://loklak-server.danghaian.com)

[mariobehling]

@zyzo Nice! loklak-client seems to be down though. Cannot test.

[zyzo]

My bad, @mariobehling could you please test again ?

[mariobehling]

Login screen for twitter stays empty - cannot login.

[zyzo]

There is an unknown problem with twitter auth indeed, but the datasource interface can currently work without logging in to twitter.

[mariobehling]

Ok, a few questions. I could see my connections without logging in.

1. Where does the data of http://cmap-fossasia-api.heroku.com/ffGeoJsonp.php come from? How is it generated?
2. What does the user have to do in "source type"? How should a user know what to do here?
3. What does a user have to do in mapping rules and tweet field?

[zyzo]

@mariobehling

1. when adding new data source (Add Data Source), the source url is sent to api/push/geojson.json. The geojson push servlet do two things : save the messages & the import profile (information about source url, created time, source type, harvesting frequence.. cf. provide push API for statical information loklak/loklak_server#35). Then the list of import profiles is retrieved via api/import.json and displayed in the data source page.
2. source_type is a necessary information when pushing messages as geojson. The assumption is that the user has a data file in geojson format, that he wants to import. In order to do that, he must know what is the source_type of the source (Twitter, Fossasia, Freifunk, ..).
3. The mapping rules are optional, but necessary, if the user file isn't compatible with loklak message format (which is twitter's like).

In short the interface is a visual way to construct push query to api/push/geojson.json (more about it here http://loklak.org/api.html#geojsonpush). The interface will change to accommodate new servlets api/push/fossasia.json, api/push/openwifimap.json, etc.

[mariobehling]

@mariobehling

1. http://cmap-fossasia-api.heroku.com/ffGeoJsonp.php

   • Whose server is it?
   • Who creates this data ffGeoJsonp.php?
   • Why do we need a server running php to offer data for loklak?
   • How can we get the data directly from the communities into github. Think of the use case "I want to add my community to the map" (e.g. https://raw.githubusercontent.com/fossasia/fossasia-communities/master/Singapore/HackerspaceSG).

2. source_type is a necessary information when pushing messages as geojson. The assumption is that the user has a data file in geojson format, that he wants to import. In order to do that, he must know what is the source_type of the source (Twitter, Fossasia, Freifunk, ..).

   I am sorry but you cannot expect the user to know all kinds of formats. Do you remember we had talked about a possible way to do this like mailchimp? We possibly want to support dozens of formats in future. Also FOSSASIA or Freifunk is just a name for nearly the same format. What does the user have to do in each "source type"? I do not understand what is import and user: I see
   a) import
   b) user
   c) FOSSASIA.

3. The mapping rules are optional, but necessary, if the user file isn't compatible with loklak message format (which is twitter's like).

   This does not answer my question. You answer the question "Are the rules optional or necessary?".

   My question is: What does a user have to do in mapping rules and tweet field? What information does he/she have to insert? What goes in each field? Examples?

[zyzo]

http://cmap-fossasia-api.heroku.com/ffGeoJsonp.php Whose server is it? Who creates this data ffGeoJsonp.php? Why do we need a server running php to offer data for loklak? How can we get the data directly from the communities in github

This is really just a json file that serves http://cmap-fossasia-api.herokuapp.com/ffGeoJson.json, nothing important (It is .php to deal with jsonp). And it is used to demo here because at the moment I could only push messages in geojson format. Now, there is /api/push/fossasia.json that can be used to push data in fossasia api format. The feature just got merged into master. As said I will accommodate this change into the dataConnect interface.

Then, user will be able to add pure fossasia api data.

I see a) import, b) user, c) FOSSASIA. What does the user have to do in each "source type"? I do not understand what is import and user.

Import and user shoudn't appear, I will remove them. They exist because SourceType was used for a different purpose SourceType.java.

What does a user have to do in mapping rules and tweet field? What information does he/she have to insert?

As explained in http://loklak.org/api.html#geojsonpush, the mapping rule (parameter map_type) is used to map fields in original data to loklak message data. For instance, if the message has a field name, that should be saved as screen_name, then you can make it a rule. More discussion about map_type :

loklak/loklak_server#55 (comment)
loklak/loklak_server#55 (comment)

[zyzo]

Interface to add fossasia api data :

If no format is specified in route parameter /addConnection/:format, or if format=geosjon, user will get following form which is a little bit different :

EDIT :

Live at http://loklak-client.danghaian.com/addConnection/fossasia_api

[mariobehling]

Currently loklak-client is not available.

If the format is not specified, we should have a validation form, that automatically matches the right format. This can be a follow up issue. Regarding the import of API data, please have a look how other services like mailchimp do this: http://kb.mailchimp.com/lists/growth/import-subscribers-to-a-list

[zyzo]

I added a setting form to let users change data format. It is opened when clicking on
the gear icon :

Automatically sniffing the right format is a nice idea ! But I think it won't be 100% reliable so we should still let user choose the format her/himself.

[mariobehling]

Ok, let's iterate and get this going in the above way first. Thanks!

[zyzo]

It is now possible to add Freifunk & FOSSASIA data via the interface. Please refer to :

• server side : Fossasia push service loklak/loklak_server#77 Wifi routers push service loklak/loklak_server#84
• client side : Implement Connect Service & Data Source Interface #205 #206 #285 Improve connect page #325 Display real data source information #394