No clean way to filter map keys or values?

[WhittlesJr]

I'm finding it kind of kludgy to try to filter out map keys or values, which is something I sometimes want to do inside of a large path without breaking out of my Specter mentality. For example, I would want something along these lines:

(select-one (filter-keys [NAMESPACE (pred= "pick-me")] ) {:a 1 :b 2 :pick-me/c 3})
=> {:pick-me/c 3}

But the best I can come up with is:

(select-one (filterer [FIRST NAMESPACE (pred= "pick-me")] ) {:a 1 :b 2 :pick-me/c 3})
=> [[:pick-me/c 3]]

or

(select-one (view #(encore/filter-keys (comp #{"pick-me"} namespace) %))  {:a 1 :b 2 :pick-me/c 3})
=> {:pick-me/c 3}

So:

• If I use filterer I need to use FIRST, but that's not a very obvious hint when trying to grok it in the context of a big path. For values, I'd use LAST and that's also strange.
• In the second example, it comes out as a vector (whereas the whole draw of Specter is that it's supposed to keep your data structure's types intact.)
• In the third example, I have to avoid Specter's pathing abilities

Am I missing something here? Or am I misguided? There's plenty of other ways I could solve this problem but this seems like something Specter should be able to do cleanly...

[nathanmarz]

You can just do:

(setval [MAP-KEYS (selected? NAMESPACE #(not= % "pick-me"))] NONE {:a 1 :b 2 :pick-me/c 3})
;; => {:pick-me/c 3}

[WhittlesJr]

Thank you for the quick response! I'm not sure how to use that approach in the context of the larger path I'm working with. Here's what I'm doing:

(def data [{:id "ID 1" :a 1 :pick-me/b nil :pick-me/c 3}
           {:id "ID 2" :a 5 :pick-me/b 6 :pick-me/c nil}])

(->> (traverse [ALL
                (collect-one :id)
                (collect-one [(filterer [FIRST NAMESPACE (pred= "pick-me")])
                              (filterer [LAST some?])
                              (transformed [ALL FIRST] (comp keyword name))])]
               data)
     (reduce (fn [m [k v]] (assoc m k (into {} v))) {}))
=> {"ID 1" {:c 3}, "ID 2" {:b 6}}

How could you write that such that v comes out as a map? Like I said, I'm also not a big fan of using "FIRST/LAST" to get keys and values respectively. Is there a better way to write this?

Or is this too trivial and should I just be using the following instead, which does the same thing (and might be simpler?):

(reduce (fn [m {:keys [id] :as point}]
            (->> point
                 (encore/filter-keys (comp #{"pick-me"} namespace))
                 (encore/filter-vals some?)
                 (encore/map-keys (comp keyword name))
                 (assoc m id)))
          {}
         data)

As I've been working with this, I found that the second (non-Specter) way is actually faster (~0.8 MS vs ~1.8 MS). So obviously I have a solution to my particular code problem, but your response will still be helpful for learning how to use Specter generally.

[nathanmarz]

This is all restructuring, so Specter's a little more verbose for this. Here's how you could do it in one path:

(into {}
  (traverse
    [ALL
     (collect-one :id)
     (transformed
       [ALL
        (multi-path
          [LAST nil? (terminal-val NONE)]
          [FIRST
           (if-path [NAMESPACE (pred= "pick-me")]
             [NAMESPACE (terminal-val nil)]
             (terminal-val NONE))]
          )]
       identity)]
    data))

[WhittlesJr]

Thank you! I didn't know about terminal-val, and it's helpful to see it in action here. That'll definitely come in handy.

The Clojure+encore implementation is still faster and more readable. Do you want Specter to be able to handle this type of restructuring more elegantly, or do you think it even should? If not, and the answer is just "different tools for different jobs," then we can close this issue. Thank you again for your help.

[WhittlesJr]

Adding some kind of filter-keys and filter-vals navigators sounds good to my unlearned brain, but I dunno.

[nathanmarz]

Specter is about targeting specific parts of a data structure and leaving the rest the same, so when you're doing a restructuring that's inherently outside the scope of the project.

[WhittlesJr]

Understood!

I've been using Specter to restructure complex, deeply nested clojure.xml/parse output into lists of flat maps, and I've found it to work brilliantly for that purpose. I've been able to cut whole namespaces out of my code, and the logic is much more elegant and grokkable now. It worked so well that I figured "this must be the purpose of the library" and I was surprised when I ran into this more trivial case that didn't work out as cleanly. It's good to have a better understanding of the project's scope now. Thanks again!

[lucywang000]

For anyone interested, this type of transformation is better solved by meander:

(require '[meander.epsilon :as me])

(def data
  [{:id "ID 1" :a 1 :pick-me/b nil :pick-me/c 3}
   {:id "ID 2" :a 5 :pick-me/b 6   :pick-me/c nil}])

(me/rewrite vd2
  [{:id !id (me/keyword "pick-me" !k) (me/some !v)} ...]
  {& ([!id {(me/keyword !k) !v}] ...)})
;; => {"ID 1" {:c 3}, "ID 2" {:b 6}}

For this case, imo the meander's version is much more readable than the specter version above provided by @nathanmarz .

And I did some perf testing with the following data, which shows both solution have almost the same performance

(def data
  (into
   [{:id "ID 1" :a 1 :pick-me/b nil :pick-me/c 3}
    {:id "ID 2" :a 5 :pick-me/b 6 :pick-me/c nil}]
   (repeat 10000 {:id "ID 2" :a 5 :pick-me/b 6 :pick-me/c nil})
   ))

Generally I find specter and meander are grealy complementary to each other.

• use specter to "select a subset of a complex nested data structure" or "nav to a given location of a complex nested data structure and do some modification"
• use meander to do things like "restructuring complex nested data" or "transform complex nested data by filtering/joining different parts of it"

Each of the two could do all these tasks (so does hand-crafted clojure code), but they have their own expertise where it could make a huge difference when used properly.