OWL Activity Tab

[ppawel]

OK, so this has been brewing for over 2 years (!) now... so I thought I'd make a pull request and get some sanity/reality checks on this code and features themselves.

In this branch you will find what I named a "activity tab". This has been briefly demo'ed to people last year (or was it 2013...?), hopefully OWL API will be back soon so it can all come together again.

Currently this new tab displays many different things:

1. List of changesets - this is done in a completely different way than with the history tab and the list always shows changesets which actually have any changes in the current view.
2. List of changes - there is no such thing as "change" in OSM API but in OWL this is a basic unit of information. A change is more or less synonymous with node/way version but in addition to object versions, changes also capture more subtle alterations like moving one node of a way - this is only recorded in the OSM API as a node version but in OWL you will see the actual change caused by moving the node.
3. Details of changes - it shows what exactly has changed (which tags, previous geometry) and also uses icons from the map key to visually identify change - so if you add/remove/move/whatever a coffee shop then you will get a proper icon in the activity tab.
4. Geometry of changes - OWL API provides vector tiles which are rendered using Leaflet GeoJSON tile layer. The user can interact with the geometry - if you click something which is highlighted on the map, you will get more information about it in the sidebar - basically the sidebar will scroll through to the position so that it shows the details of a change that you clicked on the map.

You can see an example of all this on the screenshot below.

It's not pretty, I know. It's also not very intuitive to use. It's more of a showcase for the capabilities of the OWL API - to show what information the API provides. There are many ideas how to better use the API, I've also had some feature requests (for example, ppawel#26), I think some of them definitely make sense - like inspecting a single changeset. That would probably need to be a separate "mode" from the activity tab.

Anyway, if anything, it would be good to accomplish two things with this pull request:

1. Get your feedback / suggestions on the usability, design, features and all that. I have many more ideas where to take this - I'd love to integrate the changeset commenting feature - this would give us a lovely social network-style interactive activity stream. But on the other hand, it all needs some UX love, otherwise it will probably end up as some kind of monster crammed with every feature under the sun.
2. Get this stuff code reviewed or at least get some of my assumptions validated. I mean things like new dependencies introduced - client-side templates (EJS), jQuery timeago plugin. Note that the code is pretty much WIP but of course all suggestions are welcome.

Looking forward to your comments.

[tomhughes]

I think it's far too early to be worrying about the rails integration for this - until we have stable OWL software with a stable API to write against we can't possible know what will be involved in integrating it into the web site.

[systemed]

Not a bad idea to at least think about the UI, as ideally we want the UI to lead the API and implementation rather than the other way round. That said, it'll be a lot easier to make informed comment on this if/when we can get a test instance up and running.

Some really good stuff here. My first glib thought is that we should merge 'Activity' and 'History' into one UI - for the average end-user they're the same concept.

[tomhughes]

It's very hard to comment on the UI based on a single picture though... What I can say is that it clearly needs some stylistic work to bring it more in line with other parts of the current UI.

[Zverik]

Isn't it more of less the same as http://owl.apis.dev.openstreetmap.org ?

[ppawel]

I think it's far too early to be worrying about the rails integration for this

In my opinion - yes and no. Yes because even when I created this pull request I knew it will not be merged, perhaps ever, at least not in the shape it is in. On the other hand, as you can see from the commit history, I started the integration quite early. This is because it's rather hard to work on OWL API without having any visualization of the whole thing. I figured - why make another separate project/application when I can try to get this stuff into Rails Port? Or at least kick off a discussion about what/if people want this kind of features on the main site.

Not a bad idea to at least think about the UI, as ideally we want the UI to lead the API and implementation rather than the other way round.

Exactly. I tend to whine a lot about how hard OWL is to implement, scale etc. But it really is the case here that one simple frontend feature (example: showing previous geometry) can mean tons of work, thinking, testing, optimizations on the backend side of the API. So I'd rather get some feedback / bikeshedding early and adjust the API so it's possible to implement most of the requested features.

That said, it'll be a lot easier to make informed comment on this if/when we can get a test instance up and running.

For that I need to setup OWL API on zark (or some other capable server). @zerebubuth said yesterday that he will try to get a server machine up and running for this.

Isn't it more of less the same as http://owl.apis.dev.openstreetmap.org ?

Yes, it is the rebased-to-master version of that.

[ppawel]

BTW, I forgot to mention one huge open question which touches upon every layer of this, including OWL API - what do we want to do with relations in this context? I mean, when should a relation be considered "changed" and what should appear on the activity feed. I tried not to think about relations too much because implementing it will be super depressing but what I have is this:

Example 1

Assume we have some kind of administrative relation which has ways and nodes (like capitol for example).

a. When you change only a single node belonging to this relation - should this count as a change to the whole relation, i.e. should you see administrative boundaries as changed in the activity feed?

b. When you move a single way belonging to this relation - same question - should the whole relation be considered "changed" or only the moved part?

Example 2

Assume we have some kind of nested route relation with multiple levels of nesting (relation has relations etc). No idea what to do with those...

Any thoughts? What would be the must-have functionality for relations here?

[littlebtc]

How about using augmented diffs from Overpass API instead of OWL to generate data? The API seems to work great.; but since augmented diffs from Overpass API will generate the changes on the fly, the increasing server load may be a concern.

AFAIK:

• OWL generates and stores changed tiles for each changeset, faster to query, but extremely disk usage heavy
• Overpass API generates augmented diffs on the fly for each minutely updates (so it requires to filtered by bbox and filtered out other changesets when used), system load may be a problem

But IMHO using Overpass API is faster and easier to implement since it has already a working API and some great proof of concepts.

Working example for Overpass API:
http://tyrasd.github.io/latest-changes/
http://overpass-api.de/achavi/

See also:
http://www.openstreetmap.org/user/ikonor/diary/23329

[d1g]

@ppawel IMO we want a tag-focused (history) viewer, not every-node-in-every-way-relation tool.

Simply because we have low-level tools already (overpass api, achavi to display). We lack stream of changes in overpass, but it is still possible to overcome it using historic querying.

a. When you change only a single node belonging to this relation - should this count as a change to the whole relation, i.e. should you see administrative boundaries as changed in the activity feed?

If node in question had no tags - this should be no change for user. But there special cases like relation:site where you want to count child nodes, since tags from relation propagate to its child's.

b. When you move a single way belonging to this relation - same question - should the whole relation be considered "changed" or only the moved part?

"Only moved part" would mean smaller bbox at country-scale relations. Smaller bbox is easier to understand and use.

IMO tracking of geometry-topology of country/Europe-scale relations is up to specialized validators (like http://wowik.000space.com/routes/?ckattempt=1 for road network in multiple countries). There service dedicated to admin boundaries: http://osm.wno-edv-service.de/index.php/10-osm-reports/481-countries-compare-2016-02-23

IMO focus should be more on POI-centric changestream (how to track changes of shop=* or amenity=* in single city). This is how Wikimapia history tab was functioning for 8+ years.

Avoid special cases with relations

I mean, if you ignore relations completely and make a solution that will show correct

• single object (a node, a way) history chagestream. Possibility to filter out tagless geometry changes would be a plus, see my comment on (a) above
• tag change stream in one city (for shops and such) either by bbox or by admin_level

that would be amazing to have in OSM, because we had tools for road networks and admin boundaries

[ppawel]

The "tag-focused (history) viewer" in my view would be just one possible use case of OWL - at least OWL as it was planned/partly implemented by me. It was meant to be a pretty generic service providing couple of APIs - of course main focus was on finding "changes" but there was also geometry available for every change so that would not be too far from vector tiling.

The main difference to other services (at least at the time I worked on OWL - maybe something changed since then) is that OWL would let you query data at any point in time while most services either don't deal or are not scalable to full history.

Perhaps that's actually the wrong approach to the problem because such a service would require HUGE infrastructure (meaning - lots of disk space and preferably lots of processing power), by now I think the number of changesets has basically doubled and of course continues to grow.

So the target architecture would probably be something that combines couple of larger services like Overpass (e.g. augmented diffs) AND maintains a tailored full history database itself. That would reduce the complexity and make it more maintainable/deployable and less prone to grow out of control in terms of size. Of course the trade off would be that you have a string of services that depend on each other but that's maybe easier to swallow than a monolithic monster that I implemented :-)

IMO tracking of geometry-topology of country/Europe-scale relations is up to specialized validators

Yeah but from my point of view that does not provide too much value to the regular/casual/non-technical user. Of course experts use it a lot but my idea for OWL was to really make something that would be a basis for a lot of "cool" functionalities like "social" activity stream with meaningful items in it and of course scalable to full history and including the whole world.

Avoid special cases with relations

That would definitely simplify the problem A LOT. I had my mind set on handling everything or at least preparing the database design and the code to add relations later. That resulted in a lot of complexity and unanswered questions and headaches (nested relations anyone?). So I would not recommend this approach in the future.

[pnorman]

Given the current state of OWL I suggest closing this PR.