Solved: Migrating many recipes into Mealie

[bpmartin20]

TL;DR: Convert your recipes to JSON using the structure shown in the attached "test-recipe.json". Configure the first few lines of the attached "import-json.py" and run it. See Step 5 for clean-up.

Overview

I've recently completed a migration of recipe data from another program into Mealie. As an IT professional with decades of experience, I found the migration painful and time-consuming. There's a serious lack of easy-to-find essential information out there that I hope to address in this post. I'll share the process, discoveries, and code that worked for me. I won't say it's optimal or even polished, but I would have loved to come across something like this for my own migration.

There were five main steps to my migration, starting with getting the data out of the old software in a usable form. You'll probably have a different recipe source, so your initial steps may be different. Even so, however, I think you'll find that much of what I did in the middle and later steps will still be valuable.

The steps

So here were my migration steps:

1. Export the data
2. Correct/normalize the data
3. Convert the data into Mealie-acceptable JSON
4. Import the JSON into Mealie
5. Perform Manual Clean-up

Feel free to skip or skim the early steps if they don't apply to you, and pick up later on. Here we go:

Step 1. Export the data

My old system was MasterCook v20, and exporting the data was really easy, though not quite obvious. The process for each cookbook you want to export is:

1. Go to the cookbook, so you can see the titles of all the recipes in it.
2. Type "Ctrl-A" to select all the recipes. Discovering I could export all the recipes in one operation was a huge relief, as I have hundreds of recipes and thought I'd have to do them one-by-one.
3. Type "Ctrl-M", aka "File->Export and Mail..."
4. Identify where on your computer you want it saved and fill in "Single File" with an export file name.
5. Select "MC 5+ Format (*.MX2)"
6. Click "Export"

If you're using another system, you'll have to find your own way to export your data into XML or even better, JSON.

Step 2. Correct/normalize the data

MasterCook's export is in a quasi-XML format, and it arrived as a single file. When developing a new procedure I find it easier to debug small steps rather than tackling the whole thing at once. So rather than converting 200+ recipes from a single near-XML file to JSON, I decided my next step was to split the file into individual recipes, and normalize the XML. I found both steps contained in a single tool, mx2split from J. Greely's MasterCook Tools, which you can download like this:

git clone https://github.com/jgreely/mastercook-tools

I ran mx2split on my export file from Step 1, and got 200+ clean XML recipes. For anyone who is also making this journey via XML, I've attached a sample XML (see Files below) that recipe you can refer to as an example of acceptable input for the next step. Get your XML to follow that format and you'll be in good shape.

Step 3. Convert the data to JSON

Mealie wants the data to be imported in a particular JSON format. Regardless of the format of your data at this point, this step is where you'll learn the JSON format you'll need to convert it to.

The Mealie developers are prioritizing bugs and high-demand features I'm sure, and I agree with those priorities. Still, about all they give us on the required JSON format is stating that the JSON must match the Schema.org/Recipe standard. It sounds good, but I struggled with fields that didn't work and ambiguous input formats. Later I found https://demo.mealie.io/docs (find "Recipe> Expand All" about the middle of the document), which would have saved a lot of time, but even that leaves us without any working examples or formatting details. There was a lot of trial and error here, but eventually I found most of the proper field names and format.

Below I've laid out the key elements of a recipe, along with their matching JSON field name and a very brief mention of the format they want. Better though is the sample JSON file that I've attached. For the XML folks, I've also attached the "read-xml.py" Python script I used to convert my XML files to this JSON format. (See Files below)

The key JSON fields we are interested in (to the degree that any given recipe has/needs them) are:

What	JSON element name	Content
@content		format header
@type		format header
recipe name	name	string
categories	recipeCategory	comma-separated list
yield	recipeYield	string
ingredients	recipeIngredients	list
notes	notes	list of dictionaries [^1]
prep time	prepTime	string
total time	totalTime	string [^2]
source URL	url	string
nutrition	nutrition	dictionary [^3]
instructions	recipeInstructions	list of dictionaries [^4]
rating	rating	integer (0-5)

[^1] This doesn't actually quite work. The notes do arrive in the Mealie recipe, but in the wrong place. More about that later.

[^2] This should be the total time, except that the Mealie JSON importer never imports it. Maybe I'm using the wrong name. Without documentation telling us what field names the importer is expecting, it's all just a lot of well-researched trial and error.

[^3] The nutrition in a nutrition dictionary called "nutrition". I didn't actually bring this data in as what little was in the old system was mostly wrong, so I don't really know if this works.

[^4] Instructions are imported as a list of dictionaries/hashes. Each dictionary consists of "@type" -> "HowToStep" and "text" -> the instructions for that step.

Step 4. Import the JSON into Mealie

Now that you're JSON is ready, here's the payoff step. To import your JSON into Mealie you should:

1. Gather the necessary information. You'll need:

   1. the URL for your Mealie installation (installation dependent), and
   2. an API token. To generate an API token:
      1. Log into Mealie
      2. Click Settings in the left side panel (click the hamburger menu to open it if it's closed.)
      3. Click User Settings from the pop-up menu.
      4. Under "Personal" you'll find "API Tokens". Click "Manage your API Tokens" and create a token. Copy that in to your copy/paste buffer.

2. Next, edit the attached "import-json.py" script (see Files below) and insert your URL and API token in the designated variables at the top of the script. Also adjust the JSON_FOLDER variable to match the location where your JSON recipes are stored.

3. Finally, run the script. This will do further formatting changes on each recipe to match Mealie expectations and import the data. It will also set some special boolean flags in the HTTP request (not the JSON) that Mealie is looking for. I spent days trying to figure out why my categories wouldn't import before discovering these flags. Later I discovered that the rating can't be imported during the regular import. It has to happen as a second step. The script does the second step too, but it took a long time chasing field name and format variations before I figured out that setting the rating on the first import will never work.

Step 5. Manual Clean-up

There are per-recipe things that either I couldn't solve on the import side, or just can't be done programatically. How much work you have to do depends mostly on the number of recipes you have. With a lot of imported recipes, my approach is to clean up a given recipe the first time I want to make it, and otherwise do some cleaning whenever I have a bit of free time. Here are the details:

1. Recipe Notes: As mentioned before, I never figured out how to import them as recipe notes. I identify them as "notes" during the import, and they arrive in the recipe, but somehow they appear in the recipe description instead of the notes fields. Maybe somebody can figure this out. Otherwise, when you're cleaning up a recipe, you may want to move them out of the single description field into multiple note fields.

2. Parsing Ingredients: Mealie isn't confident it parsed the ingredients correctly, so it wants you to go through and verify anything it didn't understand. When you edit the recipe, there will be a banner message about the recipe not being parsed, and a button below the ingredients labeled "Parse". Click that and verify/correct as necessary. For unfamiliar foods, it will give you a button to add that food to its database. Adding the food is the most common scenario, but sometimes a food has multiple names. If you change the name of the food (e.g. "confectioner's sugar" to "powdered sugar") it will instead give you a button to add the new name as an alias for the old name. It's the graceful handling of situations like these that tell me the Mealie developers have put a lot of thought into the software.

About seed data.

In various places including Settings->Data Management, you'll have the option to load various kinds of "seed" data. This is data gleaned from their training data source of different foods, different units of measure, recipe categories, and shopping list labels. (Think of the shopping list labels as grocery store departments, like Produce, Dairy, etc.) I highly recommend loading the "Units" seed data as it will teach Mealie about your local units of measure and their equivalences.

The "Food" seed data, however, is more debatable. I saw a number of posts from people asking how to delete the seed data, because they didn't like it. It has pros and cons. Here's the story...

If you load the Food seed data, you'll teach Mealie about a bunch of different recipe ingredients so it won't have to ask you about them the first time you parse a recipe that uses them. This makes the "Parsing Ingredients" process easier, because Mealie already recognizes a bunch of foods like salt and flour, but also like abalone. If you don't load the Food seed data, the first time you parse a recipe that uses salt, you'll have to click a button to create a new food called "salt". After that, salt is known and you'll never have to do it again, but flour is out there still waiting. So far, adding the Food seed data sounds helpful.

But the Food seed data comes with a penalty, too. Any time you start typing a food ingredient, say in a new recipe or on the shopping list, a pop-up box will appear showing you some possible foods that match. You can just click on these to save typing. For example, if you want to add bananas to the shopping list and type in the "b", Mealie will helpfully open a drop-down box of every food it knows about with an "b" in it. (Well, the top few will show with the rest if you scroll down in the pop-up box). Type the next letter, "a", and it will narrow the list to things with "ba" in the name. Without the seed data, "bananas" will probably show up right away. With the seed data loaded, however, that drop down box will include such things as "abalone", and perhaps enough other stuff that "banana" is hidden until you scroll or type more.

Loading the Food seed data creates a whole lot of noise in these situations that I find more troublesome than helpful. The parsing ingredients is a one-time process, but loading a lot of clutter into your food database is forever. Furthermore, the answer to all those posts about "how do I remove the seed data" is always "wipe the entire database or manually delete them one-by-one". I wiped my database and started over. So, if you're planning to load the Food seed data, I recommend that you take a Mealie back-up first, and be ready to restore it if you have regrets.

As for parsing common ingredients, I'm about 20 recipes in to my recipe clean-up, and on the most recent recipe I only had to teach it about two new foods because it had already seen all the other ones once. It's not bad, but it'll still be a while before I can call it finished.

Conclusion

So that's what I know. Really, most of my time was spent trying to guess what JSON names and formats to use for each part of the recipe. Hopefully my discoveries will make your migration less painful. Except for some clean-up, I'm done now. My wife and I are two weeks in and absolutely loving Mealie. Mealie is a relatively young project and there are certainly features that I imagine will be coming along as time permits. Still, given the software we were using before, switching to Mealie has been like switching from a horse and buggy to a Porsche. I hope you enjoy it as much as we do. Good luck.

Files

test-recipe.xml
read-xml.py
test-recipe.json
import-json.py