Elixir can read Marshal Format #2

gaynetdinov · 2015-12-08T22:08:36Z

Hello.

Your README says:

Since Elixir can't (yet!) read Marshal format...

This is no longer true, I've implemented Ruby Marshal format in Elixir https://github.com/gaynetdinov/ex_marshal

I'd be happy if you give it a try. Feedback is welcome.
Thanks!

cconstantin · 2015-12-09T16:41:57Z

Thanks @gaynetdinov. I'll look into it some time next week.

pragdave · 2016-01-28T05:08:05Z

I this just a question of changing the Phoenix session serializer, or are there dragons lurking?

If just the former, I could give it a try.

Dave

cconstantin · 2016-01-28T16:55:12Z

Sorry, life got in the way. @pragdave yes, it should be just a matter of changing the serializer in the plug configuration.

cconstantin · 2016-02-06T02:06:16Z

@pragdave did you get a chance to try the marshal serializer?

pragdave · 2016-02-07T03:53:19Z

Yes and no. I made a start, but then bumped into the issue that all the
encrypting instructions were for (I assume) Rails 4, and my current need is
a big Rails 3 app. I felt I was about to go down a rabbit hole, so instead
I wrote a trivial Rails endpoint that my Phoenix app can use. It takes a
session cookie and returns the contents as JSON.

Cheers

Dave

On Fri, Feb 5, 2016 at 8:06 PM, Chris Constantin notifications@github.com
wrote:

@pragdave https://github.com/pragdave did you get a chance to try the
marshal serializer?

—
Reply to this email directly or view it on GitHub
#2 (comment)
.

rmosolgo · 2016-08-16T22:08:33Z

I've taken a try at applying ExMarshal as a serializer but I'm a bit stuck.

I wrote a serializer like this:

  defmodule RailsSessionSerializer do
    def encode(value) do
      {:ok, ExMarshal.encode(value)}
    end

    def decode(value) do
      {:ok, ExMarshal.decode(value)}
    end
  end

And attached it to my endpoint like this:

  plug Plug.Session,
    store: PlugRailsCookieSessionStore,
    key: "_my_key",
    domain: ".my.domain",
    secure: true,
    signing_salt: "my-salt",
    encrypt: true,
    encryption_salt: "my-other-salt",
    key_iterations: 1000,
    key_length: 64,
    key_digest: :sha,
    serializer: RailsSessionSerializer

But sadly it seems to die during deserialization:

** (exit) an exception was raised:
    ** (ExMarshal.DecodeError) term which starts with the following symbol is not supported: "\""
        lib/ex_marshal/decoder.ex:44: ExMarshal.Decoder.decode_element/2
        lib/ex_marshal/decoder.ex:249: ExMarshal.Decoder.decode_hash/4
        lib/ex_marshal/decoder.ex:3: ExMarshal.Decoder.decode/1
        (helpdesk) lib/helpdesk/endpoint.ex:8: Helpdesk.Endpoint.RailsSessionSerializer.decode/1
        lib/plug_rails_cookie_session_store.ex:120: PlugRailsCookieSessionStore.decode/2
        (plug) lib/plug/session.ex:74: anonymous fn/5 in Plug.Session.fetch_session/1
        (plug) lib/plug/debugger.ex:185: Plug.Debugger.maybe_fetch_session/1
        (plug) lib/plug/debugger.ex:172: Plug.Debugger.render/6
        (plug) lib/plug/debugger.ex:152: Plug.Debugger.__catch__/5
        (helpdesk) lib/helpdesk/endpoint.ex:1: Helpdesk.Endpoint.call/2
        (plug) lib/plug/adapters/cowboy/handler.ex:15: Plug.Adapters.Cowboy.Handler.upgrade/4
        (cowboy) src/cowboy_protocol.erl:442: :cowboy_protocol.execute/4

I'm quite new to this, so I didn't know a good way to debug. I found ex_marshal in my deps and added:

      # decoder.ex, line 39
      symbol ->
        IO.inspect symbol
        IO.inspect to_char_list(symbol)
        IO.inspect value
        raise ExMarshal.DecodeError, reason: {:not_supported, symbol}

which gave me some output:

[EM error]
"\""
'"'
<<12, 47, 101, 118, 101, 110, 116, 115, 73, 34, 26, 115, 101, 115, 115, 105,
  111, 110, 95, 105, 110, 95, 114, 101, 100, 105, 115, 95, 104, 97, 115, 104, 6,
  59, 0, 70, 84, 73, 34, 20, 115, 116, 114, 105, 112, 101, 95, 114, 101, 100,
  ...>>

I'm not sure what's going wrong ... or where, but I thought I'd share in case anyone has encountered this before!

gaynetdinov · 2016-08-17T07:56:34Z

@rmosolgo thank you for your effort! Could you please try to put IO.inspect(value, width: 300) so the value ExMarshal failed to decode will be displayed fully(the whole list without ... in the end) so I can try to decode it using ExMarshal on my own. Thanks again!

rmosolgo · 2016-08-17T13:14:57Z

I ended up using limit:, like this:

      symbol ->
        IO.puts "[EM error]"
        IO.inspect symbol
        IO.inspect to_char_list(symbol)
        IO.inspect value, limit: 3000
        raise ExMarshal.DecodeError, reason: {:not_supported, symbol}

And it gave me:

[EM error]
"\""
'"'
<<12, 47, 101, 118, 101, 110, 116, 115, 73, 34, 26, 115, 101, 115, 115, 105,
  111, 110, 95, 105, 110, 95, 114, 101, 100, 105, 115, 95, 104, 97, 115, 104, 6,
  59, 0, 70, 84, 73, 34, 20, 115, 116, 114, 105, 112, 101, 95, 114, 101, 100,
  105, 114, 101, 99, 116, 6, 59, 0, 70, 73, 34, 32, 104, 116, 116, 112, 58, 47,
  47, 103, 105, 118, 105, 110, 103, 46, 112, 99, 111, 46, 100, 101, 118, 47,
  115, 101, 116, 117, 112, 6, 59, 0, 84, 73, 34, 35, 103, 114, 111, 117, 112,
  115, 95, 112, 101, 114, 115, 105, 115, 116, 101, 100, 95, 102, 105, 108, 116,
  101, 114, 95, 112, 97, 114, 97, 109, 115, 6, 59, 0, 70, 123, 7, 48, 73, 34, 0,
  6, 59, 0, 70, 58, 20, 108, 97, 115, 116, 95, 103, 114, 111, 117, 112, 95, 116,
  121, 112, 101, 48, 73, 34, 10, 102, 108, 97, 115, 104, 6, 59, 0, 84, 123, 7,
  73, 34, 12, 100, 105, 115, 99, 97, 114, 100, 6, 59, 0, 84, 91, 6, 73, 34, 19,
  110, 101, 120, 116, 95, 98, 114, 111, 97, 100, 99, 97, 115, 116, 6, 59, 0, 70,
  73, 34, 12, 102, 108, 97, 115, 104, 101, 115, 6, 59, 0, 84, 123, 6, 73, 34,
  19, 110, 101, 120, 116, 95, 98, 114, 111, 97, 100, 99, 97, 115, 116, 6, 59, 0,
  70, 48>>

I'm interested to hear what you find!

gaynetdinov · 2016-08-17T13:16:31Z

Ah, yes, limit. Thanks! I'll take a look.

gaynetdinov · 2016-08-17T13:48:30Z

Can you please put your debug statement IO.inspect value, limit: 3000 here https://github.com/gaynetdinov/ex_marshal/blob/master/lib/ex_marshal/decoder.ex#L3, before case. I should've asked this first, sorry. The point is that the binary you provided is not full, some parts of it might be decoded before. Thanks!

gaynetdinov · 2016-08-17T14:09:13Z

Also if you can output the original value which is going be encoded by Ruby, that would be super awesome :)

rmosolgo · 2016-08-17T15:07:18Z

Sure, here's what I've got:

Full marshaled binary

I put this a little bit sooner, so it also includes the marshal version bytes:

  defmodule RailsSessionSerializer do
    # ... 
    def decode(value) do
      IO.puts("[Endpoint] decode")
      IO.inspect(value, limit: 5000)
      {:ok, ExMarshal.decode(value)}
    end
  end

[Endpoint] decode
<<4, 8, 123, 21, 73, 34, 15, 115, 101, 115, 115, 105, 111, 110, 95, 105, 100, 6,
  58, 6, 69, 84, 73, 34, 69, 97, 55, 100, 51, 53, 56, 54, 52, 56, 102, 49, 54,
  97, 50, 101, 100, 53, 100, 54, 54, 100, 100, 50, 97, 98, 97, 55, 55, 54, 97,
  57, 57, 55, 51, 48, 52, 50, 97, 54, 102, 52, 100, 53, 49, 49, 97, 48, 101, 57,
  50, 100, 54, 98, 102, 56, 100, 48, 99, 101, 52, 55, 53, 50, 54, 6, 59, 0, 70,
  73, 34, 16, 95, 99, 115, 114, 102, 95, 116, 111, 107, 101, 110, 6, 59, 0, 70,
  73, 34, 49, 101, 86, 53, 98, 114, 74, 76, 115, 114, 80, 111, 103, 107, 76, 47,
  66, 108, 65, 74, 67, 109, 107, 76, 109, 51, 74, 49, 99, 101, 74, 102, 43, 81,
  71, 54, 97, 47, 98, 85, 66, 99, 81, 107, 61, 6, 59, 0, 70, 73, 34, 27, 97, 99,
  99, 111, 117, 110, 116, 95, 99, 101, 110, 116, 101, 114, 95, 117, 115, 101,
  114, 95, 105, 100, 6, 59, 0, 70, 105, 6, 73, 34, 36, 97, 99, 99, 111, 117,
  110, 116, 95, 99, 101, 110, 116, 101, 114, 95, 118, 101, 114, 105, 102, 105,
  101, 100, 95, 117, 115, 101, 114, 95, 105, 100, 6, 59, 0, 70, 105, 6, 73, 34,
  35, 97, 99, 99, 111, 117, 110, 116, 95, 99, 101, 110, 116, 101, 114, 95, 111,
  114, 103, 97, 110, 105, 122, 97, 116, 105, 111, 110, 95, 105, 100, 6, 59, 0,
  70, 105, 6, 73, 34, 25, 115, 104, 111, 119, 95, 98, 114, 111, 119, 115, 101,
  114, 95, 119, 97, 114, 110, 105, 110, 103, 6, 59, 0, 70, 70, 73, 34, 23, 97,
  112, 112, 95, 118, 101, 114, 115, 105, 111, 110, 95, 115, 116, 114, 105, 110,
  103, 6, 59, 0, 70, 73, 34, 8, 48, 46, 48, 6, 59, 0, 70, 73, 34, 21, 115, 101,
  115, 115, 105, 111, 110, 95, 105, 110, 95, 114, 101, 100, 105, 115, 6, 59, 0,
  70, 84, 73, 34, 14, 103, 117, 101, 115, 116, 95, 105, 100, 115, 6, 59, 0, 70,
  91, 0, 73, 34, 9, 99, 115, 114, 102, 6, 59, 0, 70, 64, 9, 73, 34, 13, 116,
  114, 97, 99, 107, 105, 110, 103, 6, 59, 0, 70, 123, 7, 73, 34, 20, 72, 84, 84,
  80, 95, 85, 83, 69, 82, 95, 65, 71, 69, 78, 84, 6, 59, 0, 84, 73, 34, 45, 100,
  99, 101, 53, 98, 101, 50, 56, 53, 98, 52, 102, 99, 52, 53, 99, 55, 53, 102,
  49, 100, 100, 52, 55, 50, 50, 52, 48, 51, 55, 50, 99, 99, 55, 48, 102, 55,
  100, 99, 48, 6, 59, 0, 70, 73, 34, 25, 72, 84, 84, 80, 95, 65, 67, 67, 69, 80,
  84, 95, 76, 65, 78, 71, 85, 65, 71, 69, 6, 59, 0, 84, 73, 34, 45, 54, 54, 101,
  97, 101, 57, 55, 49, 52, 57, 50, 57, 51, 56, 99, 50, 100, 99, 99, 50, 102, 98,
  49, 100, 100, 99, 56, 100, 55, 101, 99, 51, 49, 57, 54, 48, 51, 55, 100, 97,
  6, 59, 0, 70, 73, 34, 28, 108, 97, 115, 116, 95, 118, 105, 115, 105, 116, 101,
  100, 95, 97, 100, 109, 105, 110, 95, 112, 97, 103, 101, 6, 59, 0, 70, 34, 12,
  47, 101, 118, 101, 110, 116, 115, 73, 34, 26, 115, 101, 115, 115, 105, 111,
  110, 95, 105, 110, 95, 114, 101, 100, 105, 115, 95, 104, 97, 115, 104, 6, 59,
  0, 70, 84, 73, 34, 20, 115, 116, 114, 105, 112, 101, 95, 114, 101, 100, 105,
  114, 101, 99, 116, 6, 59, 0, 70, 73, 34, 32, 104, 116, 116, 112, 58, 47, 47,
  103, 105, 118, 105, 110, 103, 46, 112, 99, 111, 46, 100, 101, 118, 47, 115,
  101, 116, 117, 112, 6, 59, 0, 84, 73, 34, 35, 103, 114, 111, 117, 112, 115,
  95, 112, 101, 114, 115, 105, 115, 116, 101, 100, 95, 102, 105, 108, 116, 101,
  114, 95, 112, 97, 114, 97, 109, 115, 6, 59, 0, 70, 123, 7, 48, 73, 34, 0, 6,
  59, 0, 70, 58, 20, 108, 97, 115, 116, 95, 103, 114, 111, 117, 112, 95, 116,
  121, 112, 101, 48, 73, 34, 10, 102, 108, 97, 115, 104, 6, 59, 0, 84, 123, 7,
  73, 34, 12, 100, 105, 115, 99, 97, 114, 100, 6, 59, 0, 84, 91, 6, 73, 34, 19,
  110, 101, 120, 116, 95, 98, 114, 111, 97, 100, 99, 97, 115, 116, 6, 59, 0, 70,
  73, 34, 12, 102, 108, 97, 115, 104, 101, 115, 6, 59, 0, 84, 123, 6, 73, 34,
  19, 110, 101, 120, 116, 95, 98, 114, 111, 97, 100, 99, 97, 115, 116, 6, 59, 0,
  70, 48>>

Rails session

I put this:

Rails.logger.info(JSON.pretty_generate(session.as_json))

{
  "session_id": "a7d358648f16a2ed5d66dd2aba776a9973042a6f4d511a0e92d6bf8d0ce47526",
  "_csrf_token": "eV5brJLsrPogkL/BlAJCmkLm3J1ceJf+QG6a/bUBcQk=",
  "account_center_user_id": 1,
  "account_center_verified_user_id": 1,
  "account_center_organization_id": 1,
  "show_browser_warning": false,
  "app_version_string": "0.0",
  "session_in_redis": true,
  "guest_ids": [

  ],
  "csrf": "eV5brJLsrPogkL/BlAJCmkLm3J1ceJf+QG6a/bUBcQk=",
  "tracking": {
    "HTTP_USER_AGENT": "dce5be285b4fc45c75f1dd472240372cc70f7dc0",
    "HTTP_ACCEPT_LANGUAGE": "66eae971492938c2dcc2fb1ddc8d7ec3196037da"
  },
  "last_visited_admin_page": "/events",
  "session_in_redis_hash": true,
  "stripe_redirect": "http://giving.pco.dev/setup",
  "groups_persisted_filter_params": {
    "": "",
    "last_group_type": null
  },
  "flash": {
    "discard": [
      "next_broadcast"
    ],
    "flashes": {
      "next_broadcast": null
    }
  }
}

gaynetdinov · 2016-08-17T21:12:32Z

I guess I fixed the issue you've found, but I'm not sure if your rails session is now decoded properly. The problem is that you've pasted your session as JSON.pretty_generate(session.as_json) which might hide some original values(string vs symbol, nil -> null, nil -> "" and so on). For example this part:

"groups_persisted_filter_params": {
  "": "",
  "last_group_type": null
},

"":"" looks a bit fishy :) And ExMarshal decoded it as nil: "" which looks even more strange.

Anyway, can you please use fix-decoding-of-string-encoding branch of ExMarshal and try again? Also, it would be great if you can provide the output of the session variable as is.

You can find how ExMarshal decoded your rails session here https://github.com/gaynetdinov/ex_marshal/pull/7/files#diff-567c68c21ae898ac2cd38d0bfef1b6caR293

Thank you!

rmosolgo · 2016-08-17T21:38:02Z

Derp, I totally forgot how as_json would mess up the data types, sorry!

Here's a snippet from Rails.logger.info(session.inspect):

"action_dispatch.request.unsigned_session_cookie"=>{"session_id"=>"a7d358648f16a2ed5d66dd2aba776a9973042a6f4d511a0e92d6bf8d0ce47526", "_csrf_token"=>"eV5brJLsrPogkL/BlAJCmkLm3J1ceJf+QG6a/bUBcQk=", "account_center_user_id"=>1, "account_center_verified_user_id"=>1, "account_center_organization_id"=>1, "show_browser_warning"=>false, "app_version_string"=>"0.0", "session_in_redis"=>true, "guest_ids"=>[], "csrf"=>"eV5brJLsrPogkL/BlAJCmkLm3J1ceJf+QG6a/bUBcQk=", "tracking"=>{"HTTP_USER_AGENT"=>"dce5be285b4fc45c75f1dd472240372cc70f7dc0", "HTTP_ACCEPT_LANGUAGE"=>"66eae971492938c2dcc2fb1ddc8d7ec3196037da"}, "last_visited_admin_page"=>"/events", "session_in_redis_hash"=>true, "stripe_redirect"=>"http://giving.pco.dev/setup", "groups_persisted_filter_params"=>{nil=>"", :last_group_type=>nil}, "flash"=>{"discard"=>["next_broadcast"], "flashes"=>{"next_broadcast"=>nil}}}

So, it looks like you were right about nil => "" 😆

I tried the new branch, it works!! This is so awesome!!

cconstantin · 2016-08-17T21:51:52Z

@gaynetdinov thanks for following up on this. Would either you or @rmosolgo be willing to send a PR for the README to detail use of plug_rails_cookie_session_store with ex_marshal? Thanks in advance.

gaynetdinov · 2016-08-18T07:34:21Z

@rmosolgo cool! Glad it worked.

Would you like to send a PR with details how to setup ex_marshal and plug_rails_cookie_session_store to work together?

rmosolgo · 2016-08-18T18:49:21Z

👍 sure, i'm an Elixir novice but I can suggest a doc change with the code I posted above!

rmosolgo · 2016-08-18T19:08:02Z

Ok, here's my attempt! #3

gaynetdinov mentioned this issue Aug 18, 2016

Fix decoding of string encoding gaynetdinov/ex_marshal#7

Merged

rmosolgo mentioned this issue Aug 18, 2016

doc(Readme) add note about using ExMarshal #3

Merged

gaynetdinov closed this as completed Sep 8, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Elixir can read Marshal Format #2

Elixir can read Marshal Format #2

gaynetdinov commented Dec 8, 2015

cconstantin commented Dec 9, 2015

pragdave commented Jan 28, 2016

cconstantin commented Jan 28, 2016

cconstantin commented Feb 6, 2016

pragdave commented Feb 7, 2016

rmosolgo commented Aug 16, 2016

gaynetdinov commented Aug 17, 2016

rmosolgo commented Aug 17, 2016

gaynetdinov commented Aug 17, 2016

gaynetdinov commented Aug 17, 2016

gaynetdinov commented Aug 17, 2016

rmosolgo commented Aug 17, 2016

gaynetdinov commented Aug 17, 2016

rmosolgo commented Aug 17, 2016

cconstantin commented Aug 17, 2016 •

edited

gaynetdinov commented Aug 18, 2016

rmosolgo commented Aug 18, 2016

rmosolgo commented Aug 18, 2016

Elixir can read Marshal Format #2

Elixir can read Marshal Format #2

Comments

gaynetdinov commented Dec 8, 2015

cconstantin commented Dec 9, 2015

pragdave commented Jan 28, 2016

cconstantin commented Jan 28, 2016

cconstantin commented Feb 6, 2016

pragdave commented Feb 7, 2016

rmosolgo commented Aug 16, 2016

gaynetdinov commented Aug 17, 2016

rmosolgo commented Aug 17, 2016

gaynetdinov commented Aug 17, 2016

gaynetdinov commented Aug 17, 2016

gaynetdinov commented Aug 17, 2016

rmosolgo commented Aug 17, 2016

Full marshaled binary

Rails session

gaynetdinov commented Aug 17, 2016

rmosolgo commented Aug 17, 2016

cconstantin commented Aug 17, 2016 • edited

gaynetdinov commented Aug 18, 2016

rmosolgo commented Aug 18, 2016

rmosolgo commented Aug 18, 2016

cconstantin commented Aug 17, 2016 •

edited