Mongodb -> Postgres :: unsupported type mejson

[remotevision]

Directions

I am using transporter to migrate MongoDB data to Postgres. One issue I've run into is when I have a JSON array stored in Mongo. For this particular piece of data, I would like to store it as JSON in Postgres as well but get the following error.

ERROR: write message error (sql: converting Exec argument $5 type: unsupported type mejson.S, a slice of interface)  path="source-mongo/9d6e36b4-5e08-4ecf-5f19-4ab91de10d08/c2efc68a-8883-441d-4f08-69203041bd36/sink-postgres"

Example of the value it errors on:

[
  {
    "data": "https://localhost/venue/12345/header.jpg",
    "name": "header"
  }
]

System info:

• Transporter version
  v0.2.1

• OS
  Mac OSX (Sierra)

• DB version(s)
  Mongodb v2.6.3
Postgres v9.5.5

Proposal:

Add support for JSON in the Postgres adapter.

[jipperinbham]

@remotevision thanks for the details in the issue report. it looks like you're also using 2 JS transformer functions. Can you provide the pipeline.js and the JS transformer functions being used?

[remotevision]

absolutely. Here you go.

pipeline.js

Source({name:"source-mongo", namespace:"production.venues"})
.transform({ filename: "venue-remove-fields.js", namespace: "venues." })
.transform({ filename: "venue-rename-fields.js", namespace: "venues." })
.save({name:"sink-postgres", namespace:"venues./.*/"})

venue-remove-fields.js (stuff at the bottom is for troubleshooting)

module.exports = function(msg) {
  delete msg.data._id;
  delete msg.data.__v
  delete msg.data.displayName;
  delete msg.data.categories;
  delete msg.data.tags;
  delete msg.data.markets;
  delete msg.data.location;
  delete msg.data.hoursOfOperation;
  delete msg.data.twitterHashtags;
  delete msg.data.instagramHashtags;
  delete msg.data.facebookPage;
  delete msg.data.contacts;
  delete msg.data.privateNotes;
  delete msg.data.comments;

  // temp
  // delete msg.data.images;
  console.log(JSON.stringify(msg.data.images));
  delete msg.data.barcodeTypes;
  delete msg.data.phone;
  delete msg.data.partner;

  return msg;
};

venue-rename-fields.js

module.exports = function(msg) {

  msg.data.siteUrl = msg.data.siteurl;
  delete msg.data.siteurl;

  msg.data.pin_number = msg.data.pinNumber;
  delete msg.data.pinNumber;

  msg.data.barcode_types = msg.data.barcodeTypes;
  delete msg.data.barcodeTypes;

  msg.data.media = msg.data.images;
  delete msg.data.images;

  msg.data.partner_id = msg.data.partner;
  delete msg.data.partner;

  msg.data.updated_at = msg.data.modified;
  delete msg.data.modified;

  msg.data.created_at = msg.data.created;
  delete msg.data.created;

  return msg;
};

[jipperinbham]

ok, I think I see the problem, not sure I'll have time to get anything pushed up this weekend but will get to it for sure on Monday.

[jipperinbham]

@remotevision I just published v0.2.2 to see if it addresses your issue, please grab a binary from https://github.com/compose/transporter/releases/tag/v0.2.2.

[remotevision]

Awesome, I'll try it out tonight. Thanks for the quick response!

[remotevision]

@jipperinbham thanks for the quick turnaround! just wanted to let you know this worked and everything is good to go.

[jipperinbham]

@remotevision awesome! glad things are working as expected for you now.

just a heads up, we just put out v0.3.0 which was a major overhaul to running transporter, you may want to take a look at the new native transformers (omit and rename).

[remotevision]

great to see these added to v0.3.x. I'm having an issue getting my same pipeline setup with the v0.3 version however. digging in more before opening an issue.