ActionView::Template::Error (undefined method `redacted?' for nil:NilClass) in app/views/browse/_relation_member.html.erb

[gabyshev]

So I'm trying to run OSM in docker with latest master.

I downloaded and imported database for Monaco because it's small (http://download.geofabrik.de/europe/monaco-latest.osm.pbf)

I'm running it on top of phusion/passenger.

Dockerfile looks like:

FROM phusion/passenger-ruby24

COPY --chown=app:app ./osm /home/app/osm

WORKDIR /home/app/osm

RUN echo "gem 'tzinfo-data'" >> Gemfile
RUN bundle install

RUN apt-get update && apt-get install -y libv8-dev osmosis postgresql-client

# needed for successful assets precompile
RUN cp config/example.application.yml config/application.yml
RUN cp config/example.database.yml config/database.yml

RUN bundle exec rake i18n:js:export
RUN bundle exec rake assets:precompile

RUN chown -R app:app /home/app/osm/tmp

RUN rm -f /etc/service/nginx/down
RUN rm /etc/nginx/sites-enabled/default
ADD osm.conf /etc/nginx/sites-enabled/osm.conf
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

RUN curl -o /home/app/monaco.osm.pbf http://download.geofabrik.de/europe/monaco-latest.osm.pbf

ADD ./docker/Application/bin/configure.sh /etc/my_init.d/configure.sh
ADD ./docker/Application/bin/migrate.sh /home/app/migrate.sh
RUN chmod +x /etc/my_init.d/configure.sh && chmod +x /home/app/migrate.sh

CMD ["/sbin/my_init"]

configure.sh configures database.yml and application.yml and migration.sh create functions, extensions and migrate data using osmosis:

osmosis --read-pbf /home/app/monaco.osm.pbf --log-progress --write-apidb database=$POSTGRES_DB host=$POSTGRES_HOST:$POSTGRES_PORT validateSchemaVersion=no user=$POSTGRES_USER password=$POSTGRES_PASSWORD dbType=postgresql

I see an error when trying to open relation /relation/2220322:

Started GET "/relation/2220322" for 172.27.0.1 at 2018-09-10 06:23:11 +0000
Processing by BrowseController#relation as HTML
  Parameters: {"id"=>"2220322"}
  Rendering browse/feature.html.erb within layouts/map
  Rendered collection of browse/_tag.html.erb [145 times] (23.5ms)
  Rendered browse/_tag_details.html.erb (30.8ms)
  Rendered browse/_common_details.html.erb (43.1ms)
  Rendered collection of browse/_relation_member.html.erb [36 times] (70.3ms)
  Rendered browse/_relation.html.erb (120.0ms)
  Rendered browse/feature.html.erb within layouts/map (127.2ms)
Completed 500 Internal Server Error in 322ms (ActiveRecord: 51.4ms)

ActionView::Template::Error (undefined method `redacted?' for nil:NilClass):
    1: <%
    2:   member_class = link_class(relation_member.member_type.downcase, relation_member.member)
    3:   linked_name = link_to printable_name(relation_member.member), { :action => relation_member.member_type.downcase, :id => relation_member.member_id.to_s }, :title => link_title(relation_member.member), :rel => link_follow(relation_member.member)
    4:   type_str = t '.type.' + relation_member.member_type.downcase
    5: %>

app/helpers/browse_helper.rb:35:in `link_class'

The error happening in app/views/browse/_relation_member.html.erb because helper method app/helpers/browse_helper.rb trying to call redacted?:

  def link_class(type, object)
    classes = [type]

    if object.redacted?
      classes << "deleted"
    else
      classes += icon_tags(object).flatten.map { |t| h(t) }
      classes << "deleted" unless object.visible?
    end

    classes.join(" ")
  end

_relation_member.html.erb rendered by _relation.html.erb:

    <% unless relation.relation_members.empty? %>
      <h4><%= t '.members' %></h4>
      <ul><%= render :partial => "relation_member", :collection => relation.relation_members %></ul>
    <% end %>

But in the console I found that not every relation_member has a member:

Loading production environment (Rails 5.2.0)
2.4.4 :001 > r = Relation.preload(:relation_tags, :containing_relation_members, :changeset => [:changeset_tags, :user], :relation_members => :member).find("2220322")
 => #<Relation id: 2220322, changeset_id: 0, timestamp: "2018-05-04 10:30:32", visible: true, version: 25>
2.4.4 :002 > r.relation_members.empty?
 => false
2.4.4 :003 > r.relation_members[25].member
 => nil
2.4.4 :004 > r.relation_members[0].member
 => #<Relation id: 5986436, changeset_id: 0, timestamp: "2016-05-18 07:21:05", visible: true, version: 3>

Which looks like a bug to me. I would be happy to send PR to fix it but I have no idea what is the logic should be.

Shall we just skip relation_members without members ?

[gravitystorm]

The extracts provided by Geofabrik are not "referentially complete" - that is, they contain relations with references to members which are not included in the extract. For example, the relation that you mention (2220322) has way 398377196 as a member, but that way is not included in the extract.

The api and website code is not designed to cope with the database being in an inconsistent state - that's why you get the server error.

The way to fix this is to pre-prepare the extract to avoid having any referential integrity problems. I'm not sure off the top of my head if there's a script available for this - suggestions welcome!

[gabyshev]

@gravitystorm do I understand correctly that OSM is not supposed to work only on certain countries, but the whole planet?

[tomhughes]

No that's not what he said. What he said is that it assumes the data is referentially correct - there are a number of ways you could load a single country and have if meet that definition:

• You could load those objects outside the country that objects in the country refer to
• You could remove references from objects in the country to other objects that aren't present

You have used an extract which doesn't do either of those things (because they're hard to do) and hence you have a problem.

In general the API database does not really make any special attempt to support imports at all - it assumes that you will start an empty database and that it will be in control of ensuring that referential integrity is maintained. There are, to my knowledge, no tools for easily importing a referentially correct extract.

[woodpeck]

One way to make a given extract referentially complete is:

osmosis --read-pbf myfile.osm.pbf --bb left=-180 right=180 top=90 bottom=-90 clipIncompleteEntities=true --write-pbf mynewfile.osm.pbf

or you could simply add the --bb into your existing Osmosis pipeline. This will simply remove missing nodes from a way (though Geofabrik extracts are usually reference complete with regards to ways) and missing nodes/ways/relations from a relation. Of course the resulting objects will then not necessarily correspond to what's on OSM itself.

The other method is creating a referentially complete extract to begin with. by starting from a planet file and cutting out the area of interest using either Osmosis with complete-relations=true and complete-ways=true or the faster approach with osmium --extract using the "smart" strategy with -S types=any.

[gabyshev]

@woodpeck works like a charm. Thanks!

[liran]

@gabyshev I am currently working on this docker container. Could I refer to your configure.sh. Thanks!