No documentation for associations/foreign key joins

[limhoff-r7]

The roadmap says that associations should be supported, but I can't find any documentation on how to do more than one model in one repository in any of the examples. How do I declare associations and/or foreign keys for joins? The last code I could find was this: solnic/rom-relation@38136a8 , but there's no explanation of what I use in place of has from before that commit. Are association still pending work? I would like to use rom to allow in-memory emulation of https://github.com/rapid7/metasploit_data_models when the user doesn't have access to and/or want to setup postgres. If you feel that rom is missing too many features at this time to do this, please let me know.

[solnic]

Hey sorry for a late reply - ROM doesn't support relationship DSL but it does support joining relations. I will come up with some examples that will (hopefully) work so you should be able to judge yourself if it's good enough for you.

[limhoff-r7]

No big rush, I tried using ROM with the axiom-memory-adapter and emulating ActiveRecord::Base.first_or_create! by using ROM::Relation#restrict in a spec and 6 examples took 71 seconds, so I'm not going with ROM for now and just manually optimizing searches with hashes, etc.

It appeared that doing the check for collisions with restrict was adding 60 seconds on top of just creating the rows. Here's the code I was using. If I'm doing something incredibly stupid please let me know as I'd much prefer to be able to use ROM and an AREL-like syntax than have to manually do the in-memory search when the user doesn't have a database and I can't use AREL.

BITS_PER_BYTE = 8
RANDOM_ID_BYTES = 16
RANDOM_ID_BITS = RANDOM_ID_BYTES * BITS_PER_BYTE
RANDOM_ID_N = 2 ** RANDOM_ID_BITS

#
# Methods
#

# An environment for in-memory models.
#
# @return [ROM::Environment] environment with schema and mapping already
#   declared
def self.environment
    environment = ROM::Environment.setup(
            memory: 'memory://metasploit-framework'
    )

    environment.schema do
        base_relation :architectures do
            repository :memory

            attribute :id, Integer
            attribute :abbreviation, String
            attribute :bits, Integer
            attribute :endianness, String
            attribute :family, String
            attribute :summary, String

            key :id
        end
    end

    environment.mapping do
        architectures do
            map :id,
                    :abbreviation,
                    :bits,
                    :endianness,
                    :family,
                    :summary

            model Metasploit::Framework::Architecture
        end
    end

    seed(environment)

    environment
end


# Generates a random ID with the same number of bits as
# `SecureRandom.uuid`
#
# @return [Integer]
# @see SecureRandom.random_number
def self.random_id
    SecureRandom.random_number(RANDOM_ID_N)
end

def self.seed(environment)
    ROM::Session.start(environment) do |session|
        Metasploit::Model::Architecture::SEED_ATTRIBUTES.each_with_index do |attributes, i|
            relation = session[:architectures].restrict(attributes)

            if relation.count == 0
                seed = Metasploit::Framework::Architecture.new(attributes)
                seed.id = random_id
                seed.valid!

                # XXX not sure why I need to manually call track since the
                # rom-rb.org shows me just using save.
                session[:architectures].track(seed)
                session[:architectures].save(seed)
            end
        end

        session.flush
    end
end

[solnic]

Oh wow that's great feedback thank you for this. I will take a closer look and get back to you asap.

[schmurfy]

while somewhat offtopic your can create an inmemory sqlite database: https://gist.github.com/schmurfy/6378497

[mereghost]

If you are on *nix you can create the sqlite db on the /tmp folder, since
it is RAM based.
Em 29/08/2013 11:05, "Julien Ammous" notifications@github.com escreveu:

while somewhat offtopic your can create an inmemory sqlite database:
https://gist.github.com/schmurfy/6378497

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-23491614
.

[limhoff-r7]

metasploit-framework used sqlite in the past, but metasploit-framework is threaded and sqlite kept having issues, so I was advised that it wasn't worth using sqlite @schmurfy. Additionally, the full schema from metasploit_data_models using postgres only features like INET columns.

metasploit-framework needs to run on Windows and *nix-like systems (Linux and OSX mostly), so I can't depend on that behavior @mereghost.

[limhoff-r7]

@solnic I wanted to make sure you noticed, but I had to make two non-obvious changes from the examples on rom-rb.org:

1. I had to call session[:archtectures].track(seed) before I could call session[:architectures].save(seed).
2. I had to use session.flush instead of session.commit.

[solnic]

Yeah that's my bad. I will fix the examples. Thanks for pointing this out.

On Thu, Aug 29, 2013 at 4:24 PM, Luke Imhoff notifications@github.com
wrote:

@solnic I wanted to make sure you noticed, but I had to make two non-obvious changes from the examples on rom-rb.org:

1. I had to call session[:archtectures].track(seed) before I could call session[:architectures].save(seed).

2) I had to use session.flush instead of session.commit.

Reply to this email directly or view it on GitHub:
#3 (comment)

[solnic]

@limhoff-r7 I updated the example.

You can use a short-cut syntax to create new object and track it automatically via session[:architectures].new(seed).

How many objects do you have in that seed btw?

[solnic]

@limhoff-r7 can you see if it gets a bit faster when doing environment[:architectures].restrict(seed).count instead of using session?

The difference is that session sets up IdentityMap and state tracking and "plain" environment does not. You might get a perf boost. We also don't have an optimised #count yet so at the moment it loads all the objects matching criteria.

Is this code publicly available? I could try it out locally and see how I could improve stuff :)

[solnic]

@limhoff-r7 oh and btw how many objects do you have there?

[limhoff-r7]

16 objects when I did the test.

[limhoff-r7]

I wouldn't have used count if there was an exists? or exist? method.

[solnic]

@limhoff-r7 ugh 16 objects? that's very weird. I need to try to reproduce this, something is wrong. I was expecting you'd say 16k objects or something heh

[limhoff-r7]

I pushed up a branch with the specs running slow here: https://github.com/limhoff-r7/metasploit-framework/tree/feature/rom (so the feature/rom branch of my fork of metasploit-framework).

git clone git@github.com:limhoff-r7/metasploit-framework.git
cd metasploit-framework
git checkout feature/rom
rspec spec/app/models/metasploit/framework/memory_spec.rb

The specs as a whole will fail since this branch was cut off of feature/module-caching, which is under going major rework, so only run this one spec.

[limhoff-r7]

If you're not using an IDE like Rubymine that allows you to click-through to definition, it may not be obvious what the seed attributes actually are, so here's a direct link to them in metasploit-model: https://github.com/rapid7/metasploit-model/blob/feature/module-caching/lib/metasploit/model/architecture.rb#L51

[solnic]

@limhoff-r7 thank you! I'll investigate during the weekend

[solnic]

@snusnu do you think it makes sense to show how you can set up joined relations? it would be pretty verbose and people might be confused by it (maybe?). wdyt?

[snusnu]

@solnic yes i think that makes sense, and i don't think that it's pretty verbose by the way. Imo we shouldn't only think of this as a relationship replacement, but as demonstrating a way to map SQL views (with the added benefit of being writable under a lot of circumstances). Furthermore, I have the feeling that apps developed with ROM will typically involve some mappings of this kind. I think that quite a few places will emerge, were modeling a specific set of data is more naturally expressed by means of (mapping) a view , than by (building new objects around) relationships.

[solnic]

@snusnu I wholeheartedly agree there’s only one caveat right now - mappers don’t support loading data from a joined relation

[snusnu]

@solnic oh heh, i forgot. Do EVs work already, or is it just ECs that don't work atm?

[solnic]

@snusnu you'd have to build your own mapper, there's no EV/EC concept in rom-mapper yet but we could probably easily add it

[snusnu]

@solnic then i guess we should show an EV example with a simple custom mapper. For EC, i guess we'd either postpone it until group/ungroup lands in axiom at which point it will be trivial for rom-mapper. Or we try to come up with some intermediate solution that knows how to hand a set of flat tuples to a mapper instance for consumption. I'd hate to do that fwiw, and that's also the reason why I made a case for having group/ungroup support in axiom sooner rather than later. It's imo even more important than SQL write support, as i've said before.

[snusnu]

@solnic our recent design change, resulting in mapper only ever seeing one tuple, makes group/ungroup even more important.

[snusnu]

@solnic fwiw, that doesn't mean i'm against providing something that can be used in ROM::Relation#each and "collapses" the n flat tuples into one grouped tuple, sending that to mapper.load. But I'm afraid that we will need quite some infrastructure for that to work out reasonably well (and I haven't yet thought about write support). We'd need FK information in order to perform group properly ourselves. Unless I'm missing something, of course.

[snusnu]

@solnic I'm afraid that hand rolling group/ungroup in rom-relation, will add tons of completely superfluous complexity, given that it will be replaced by something really trivial eventually. I'm totally not sure if that's worth it.

[solnic]

Who’s talking about group/group? :) I’m talking about mapping a bunch of attributes into an embedded object.

[snusnu]

@solnic yeah, no problem for EV. but how to do EC without group/ungroup?

[snusnu]

@solnic let's continue this in IRC (needn't be now)

[dkubb]

FYI, Axiom::Relation now supports #group,#ungroup, #wrap and#unwrap`. This should open up EC and EV support in ROM once the following is added:

• ROM::Relation#group
• ROM::Relation#ungroup
• ROM::Relation#wrap
• ROM::Relation#unwrap