emigration affects too many dwarves and doesn't send dwarves away

[LurkerZee]

I was excited to use the emigration feature. I'd get rid of those pesky near-depressed/angry dorfs. Or so I thought.

I had a new fort and 1 migrant wave, dorfs were angry at the level of the second season. I used emigration enable in the hopes of removing one or two yellow-arrowed fellows. I was left with 4 dorfs: my two miners (thankfully), a hauler and an intelligent undead that I recruited through "tweak makeown".

No big deal I thought, just get a new batch. I used migrants_now and sure enough, the next day fresh faces were coming to the fort.

Next month, I was down to four or five again. The rest had left.

No big deal, I thought again, get a new batch. So I repeated this several times, apparently with no adverse effects. Except...

I noticed on Others that I had new visitors. No big deal, I thought. There were just 5... then 10... then 30... I realized after a few months that migrants that were set to emigration... stayed in my fort as "friendly", eating my FPS and otherwise doing nothing. A couple of miners were even sharing the same bed/room I assigned them while they were part of my fort, even if they did nothing and I couldn't give them any orders!

Not an insurmountable thing, I assumed. I'll at least make those two miners my own again.

I use emigration disable.

Tweak makeown.

Nothing.

Tweak makeown.

Nothing.

So as you can see, the emigrate issue is kind of bugged. Or kind of extreme. When I used it, I expected it that it'd relieve my fort of the most troublesome elements, those ready to snap. And that it'd take some time. Instead, even void dorfs left me except for 3 or 4. This option needs some heavy tweaking before it'll be actually usable.

PS To make it 3 for 3, this is not the first time I had an issue with tweak makeown. The first time was of someone else's save where someone who'd been "Visiting" yearly one retired fort, then another, then back again and so on came to the fort I made as "Visitor". I used tweak makeown on him, but it did nothing. I think he eventually left my fort.

[lethosor]

Not sure I understand - what exactly is the issue with migrants-now? It just requests a migrant wave from DF, which is subject to the same constraints that DF would apply to a regular wave. Could you clarify whether emigrating units are actually leaving the map? Is the issue that units that leave the map are returning as migrants, or something else? If it's that, it's most likely due to emigration not preventing the units from returning, which would be a duplicate of #1409 (I think) and unrelated to migrants-now.

The tweak makeown issue might be related to #1155, but I'm not sure.

[LurkerZee]

I'm sorry if I expressed myself wrongly. migrants_now has no issue by itself, it works perfectly and as I expected it would. I put it in the title because it got caught in the tangle of "emigration enable" and the buggy dwarves that wouldn't leave.

The actual bug is that "emigration enable" doesn't send dwarves away, they keep them on the map with the "Friendly" tag. I suspect that it's related to the diplomat not leaving, but I'm not sure/I haven't checked.

The actual issue that started all this is that "emigrate enable" insta-empties my fort of even void dwarves that barely stayed 2 seasons and that the command deduces should be removed. But the bug is bad too - the fact that those who should emigrate instead stay around as "Friendly" users freeloading and eating my FPS.

I haven't mentioned that the "Friendly" units that were supposed to leave but hadn't were initially hanging out around the starting wagon, because I hadn't gotten around to dismantle it. When I did dismantle it, they didn't leave, they bee-lined to my fort's dining room, telling me they're here to stay. Hopefully, until the Diplomat leaves, but I'm not holding my breath.

As an offtopic, I was hoping to use migrate enable as a less cheaty alternative to remove-stress, but if it's going to empty my fort every time I use it, I'd rather return to remove-stress. At least it was a test fort that I only used to see what's under a location without using the reveal (and possibly killing my computer) and for other (pure vanilla) testing purposes, so I'll just retire and continue my (vanilla) testing. I'm underlining vanilla here to mention that I haven't modded the game in a way to enable these glitches as far as I know.

Thanks for the answer and for the request for clarification, if you need anything more to understand, I'm more than willing to answer.

Edit: Here are all my "Friendlies" all in a row. https://imgur.com/XAaFP5H

[lethosor]

Ah, so they're not leaving at all - I wasn't sure from your first couple paragraphs. I retitled the issue - does that work?

Out of curiosity, what announcement do you get when these dwarves attempt to leave? It looks like there are a couple possibilities. For instance, sometimes they will "leave with the merchants", which probably wouldn't actually happen until the merchants leave the map.

[LurkerZee]

Ah, it says a diplomat left unhappy. It spams those every IRL-5 seconds actually. For. Every. Single. One. That. "Leaves." Honestly though, that was the least of my memories on the subject, though it's perhaps also something that someone might want to look into.

The title works, though as I said, I have as big an issue with the fact that "emigration enable" empties my fort instead of just removing the most troublesome dwarves.

[lethosor]

Ah, it says a diplomat left unhappy. It spams those every IRL-5 seconds actually

To clarify, I'm asking about these announcements created by the script. They should show up in white text. I think the ones you described are created by DF, but might be a side effect of the script.

[LurkerZee]

Perfect title, thank you!

No, the announcement is white text on an entire black "window?" saying "a diplomat has left unhappy" each time a dwarf "leaves". I click enter and 5 second later another identical message appears until the number of messages coincides with the number of dwarves that "leave". However, before those messages appear, I'm already left with 4 dwarves, but those messages continue on for minutes until, as I said, it gives one of those messages for each dwarf that was supposed to leave.

I have received none of those three messages.

I'll check the save to be certain, I retired the fortress, but kept the latest autosave.

I checked, yes, in the "a" log, it does say for every dwarf "has followed the diplomat".

So I counted each one, I have 32 "Friendlies" and 33 messages, this reminds me that I used "tweak makeown" on a Woodcutter (Woodworker? Carpenter? A yellow fellow) and it DID work. Not on those miners though for some reason. I'm willing to bet I saw 33 "A diplomat left unhappy" messages while playing the game, too.

Also, the first one with the message "has followed the diplomat" is from 24th Timber and they're still with me on the 1st of Granite next year, so they had more than a season to leave.

PS: I now tried "tweak makeown" on some of the other dwarves, no luck. It looks like that Woodworker was the exception/a fluke that it worked.

[lethosor]

Ok, those announcements are produced by DF, then - emigration only produces white announcements that show up at the bottom of the screen (and also in the a screen, as you mentioned). "Has followed the diplomat" indicates that it is indeed attempting to make the dwarves leave with any diplomats, so I'm curious if disabling that strategy works any better. Unfortunately it can't be configured without modifying the script, but here's a patch that would disable the "follow the diplomats" strategy (replace the red lines starting with - with the green ones starting with +, leaving those characters out):

diff --git a/emigration.lua b/emigration.lua
index 5cc95d5..6a54d28 100644
--- a/emigration.lua
+++ b/emigration.lua
@@ -45,10 +45,10 @@ function desert(u,method,civ)
         line = line.."joined the merchants"
         u.flags1.merchant = true
         u.civ_id = civ
-    elseif method == 'diplomat' then
-        line = line.."followed the diplomat"
-        u.flags1.diplomat = true
-        u.civ_id = civ
+    -- elseif method == 'diplomat' then
+    --     line = line.."followed the diplomat"
+    --     u.flags1.diplomat = true
+    --     u.civ_id = civ
     else
         line = line.."abandoned the settlement in search of a better life."
         u.civ_id = -1

It looks like tweak makeown should be able to undo the effects of emigration, though. If you have a dwarf exhibiting the issue, the output of these three commands would be helpful (select the problematic unit in the UI first) - ideally before and after running tweak makeown, but just after would work as well.

lua ~unit
lua ~unit.flags1
lua ~unit.flags2

[LurkerZee]

Honestly, I'd rather let the people who work DFHack handle the issues, just let me know if possible when the issues are resolved, I'm not that great at coding. As far as I know, the problem is replicable (if it's not, then it's my save's fault, then I wasted everyone's time and I apologize).

Thanks for all the help so far, hopefully this option will be improved in next versions of DFHack. Maybe I'll try it then and give you feedback.

[PatrikLundell]

I wouldn't use that script as it doesn't seem to be doing all it should do.

I don't see anything that removes them from being citizens, only (potentially) changing to another civ. I also have no idea what happens when a mother carries her baby away (assuming she'd actually leave, of course).

It can be noted that the behavior seen on the units that won't leave is similar to the one seen on bugged performance troupe members arriving after some of them became residents due to a performance group petition, as well as "returning" characters from conquered sites. However, those just hover at the embark edges until killed by invaders, rather than moving to the tavern. I don't know how to fix those either (and setting the head for the forest flag doesn't work on them).

It can also be noted that setting the "head for the forest" flags works for getting rid of visitors (not residents) only if they're not performing a "job", and socializing at the tavern is a job that may never end if the flag is set. It could be that the leave countdown improves the behavior, though.

It also looks to me as if the desertion logic is backwards: the higher the result from desireToStay the greater the probability that the random value is lower, so I think "<" should probably be ">".

[lethosor]

Honestly, I'd rather let the people who work DFHack handle the issues, just let me know if possible when the issues are resolved

It's not a high priority from our end (and from PatrikLundell's comment, the "easiest" fix may be to remove the script entirely), so I would appreciate any help in diagnosing the issue. It can take a significant time to get a save into a state where the issue occurs, and you already have one.

My instructions earlier were intended to be a way to disable a specific part of the script to see if it was causing bugs. Perhaps this is easier: try just removing these four lines from hack/scripts/emigration.lua (they start around line 48):

    elseif method == 'diplomat' then
        line = line.."followed the diplomat"
        u.flags1.diplomat = true
        u.civ_id = civ

If you can't get this to work, attaching a save from right before the issue occurs could help - ideally a small one (it needs to be under 25MB to fit within GitHub's limits, although hosting it somewhere else could also work).

[LurkerZee]

I think I found a slow and probably dirty way to fix the "tweak makeown not working on visitor". The steps (I haven't replicated it yet, but it should work) are:
Write "tweak makeown" (does nothing, right?) Unpause, pause. Write gui/gm-editor -> flags2. Modify here: visitor to false, resident to true (not sure if the latter is necessary). Escape from the menu, pause, unpause. Nothing happens. Run tweak makeown again. This makes it your unit again.

I replicated it, looks like I didn't need to modify resident (it reverts back to false anyway), unit became mine immediately after I removed the visitor flag. Hope this helps.

Replicated it again, it seems as soon as you manually remove the visitor flag, it automatically becomes your unit.

Maybe someone who knows the coding can implement the removal of the flag directly into the script? Thanks.

[lethosor]

So setting visitor = false helps with this? That could be done similar to here. I'll see if I have a save with a visitor handy.

[myk002]

Is this still an issue or was it fixed by the linked PR?

[wsfsbvchr]

Doesn't seem to work for me. The dwarves set to emigrate leave, but return immediately. Doesn't matter if they leave with merchants or diplomat, or just abandon the settlement in search of a better life, in all cases they just return immediately. Looks like the changes to their flags are reset, etc.

[lethosor]

What DFHack version are you using? There are likely to be differences between 0.47 and 50.x

[wsfsbvchr]

0.47.05-r5 atm

[myk002]

It looks like emigration logic was fixed in 0.47.05-r7. You might want to upgrade DFHack to -r7 or -r8: https://github.com/DFHack/dfhack/releases?page=2

[wsfsbvchr]

It was just a flip of a < to > in the logic that is used to determine whether to run it on a certain dwarf, but the actual mechanism for emigration ( function desert() ) doesn't seem to work. I tried even forcing it by creating a modified version of the script where it runs on a selected dwarf, to see what happens. In every test so far the dwarves leave as intended, but return immediately. Even if doing so exceeds the strict population cap. So it's as if they were on a mission or so.

[wsfsbvchr]

So, if you, for example, start a new fortress, and then immediately do misery enable and emigration enable, what happens? Does it work for you guys somehow? Based on my tests I can't say that it doesn't work at all, because I guess it could be something situational that I haven't been able to avoid.

Also, expelling seems to work, so one way to fix this could be by observing what exactly happens when you expel a citizen, and then imitating that to an extent.

[wsfsbvchr]

I modified devel/query to write to file instead of printing, and saved the outputs of these queries before and after expelling an unit:

[DFHack]# cust/query -unit -maxdepth 4
[DFHack]# cust/query -table df.global.ui.main.fortress_entity -maxdepth 4
[DFHack]# cust/query -table df.global.ui.main.fortress_entity.hist_figures -maxdepth 4
[DFHack]# cust/query -table df.global.ui.main.fortress_entity.hist_figures[27] -maxdepth 4
[DFHack]# cust/query -table df.global.world.history.figures[359] -maxdepth 4
[DFHack]# cust/query -table df.global.world.history.events -maxdepth 9 -maxlength 10000

Looks like a bunch of stuff changes, but I guess the relevant parts are that the unit is removed from df.global.ui.main.fortress_entity.hist_figures, and the relevant histfig_entity_link_memberst at df.global.world.history.figures[359].entity_links is changed to histfig_entity_link_former_memberst. There's also an event for the expelling.

So perhaps the script would work if it also removed the dwarf from the list in fortress_entity, and it'd make sense to change the entity link too, and maybe a new one if you want to force them to settle somewhere. There could also be an event describing the moving.

I'll try to do that myself when I have time, until then just leaving the findings here.