Add support for editing Stardew Valley maps #1560

Closed
bjorn opened this Issue May 2, 2017 · 16 comments

Comments

Projects
None yet
2 participants
@bjorn
Owner

bjorn commented May 2, 2017

Stardew Valley maps are stored inside .xnb files (produced by Microsoft XNA Game Studio) in .tbin format (a binary tile map format used by tIDE). It should be possible to write a Tiled plugin that can directly load and save such maps.

On the Tiled forum somebody reported to have successfully extracted the necessary .tbin and .png files and then used tIDE to open the map and save it as .tmx so that it could be opened in Tiled. It would be cool to simplify this process by supporting .tbin and maybe even .xnb files containing .tbin files directly.

@bjorn bjorn added the feature label May 2, 2017

@spacechase0

This comment has been minimized.

Show comment
Hide comment
@spacechase0

spacechase0 Jun 2, 2017

Contributor

Some other things:

  • Most (all?) of the game's XNBs are compressed (using LZX), which SimpleXNBDemapper can't handle. I did recently find this. I tested it on Data/eventConditions.xnb and it appears to have worked successfully. Their C# decompression is mostly just a port of libmspack. (It says dual-licensed LGPL/MS-PL. Not sure how that works since libmspack is only LGPL.)
  • The project above (and the original libmspack) doesn't support compression. This isn't a huge issue though, since (as SimpleXNBDemapper mentions) XNBs are valid even when not compressed.
  • We're currently using a modified version of tIDE to export a .tbin to .tmx. (Mentioned and briefly explained here). I don't know the full details of the modifications but I believe @Pathoschild was given the source code.
  • Stardew Valley maps heavily rely on individual tile properties, which it seems like Tiled doesn't support? (Or I'm just missing them.) Right now we have an object called "TileData" precisely on each tile that has properties.
  • .tbin animations only support a single frame interval per set. I don't know how our tIDE modification handles different animation lengths.
  • The .tbin tilesheet textures don't even list an extension, just "paths" or "spring_outdoorsTileSheet", due to the way the XNA content pipeline works (I think).
  • The current modding API does allow us to make maps using both our own tilesheets (which can be .png if you include the extension in the image source) or the base game tilesheets. So it'd be reeeeeeeeeeeeeally handy if we could have multiple search directories for tilesheets.

If recompression is extremely important to you, I have started writing a LZX de/compressor for some of my own projects, and I'm just going to throw it under MIT (if I finish it).

That's all I've got for now.

Contributor

spacechase0 commented Jun 2, 2017

Some other things:

  • Most (all?) of the game's XNBs are compressed (using LZX), which SimpleXNBDemapper can't handle. I did recently find this. I tested it on Data/eventConditions.xnb and it appears to have worked successfully. Their C# decompression is mostly just a port of libmspack. (It says dual-licensed LGPL/MS-PL. Not sure how that works since libmspack is only LGPL.)
  • The project above (and the original libmspack) doesn't support compression. This isn't a huge issue though, since (as SimpleXNBDemapper mentions) XNBs are valid even when not compressed.
  • We're currently using a modified version of tIDE to export a .tbin to .tmx. (Mentioned and briefly explained here). I don't know the full details of the modifications but I believe @Pathoschild was given the source code.
  • Stardew Valley maps heavily rely on individual tile properties, which it seems like Tiled doesn't support? (Or I'm just missing them.) Right now we have an object called "TileData" precisely on each tile that has properties.
  • .tbin animations only support a single frame interval per set. I don't know how our tIDE modification handles different animation lengths.
  • The .tbin tilesheet textures don't even list an extension, just "paths" or "spring_outdoorsTileSheet", due to the way the XNA content pipeline works (I think).
  • The current modding API does allow us to make maps using both our own tilesheets (which can be .png if you include the extension in the image source) or the base game tilesheets. So it'd be reeeeeeeeeeeeeally handy if we could have multiple search directories for tilesheets.

If recompression is extremely important to you, I have started writing a LZX de/compressor for some of my own projects, and I'm just going to throw it under MIT (if I finish it).

That's all I've got for now.

@spacechase0

This comment has been minimized.

Show comment
Hide comment
@spacechase0

spacechase0 Jun 9, 2017

Contributor

So I got my XNB loader working for most of the Stardew files (and certainly the ones Tiled would need). It's here. I re-ported the library I mentioned earlier here.

If you aren't happy with that license for that compressor, someone apparently wrote their own here. (It doesn't have a license right this moment, but the author tells me he'll put one up in the next day or two. He mentioned "MIT or whatever" earlier today.) (GPL)

I'm probably going to be writing a tbin loader soon to feature-creep my own project. I'll throw that under MIT as well.

Contributor

spacechase0 commented Jun 9, 2017

So I got my XNB loader working for most of the Stardew files (and certainly the ones Tiled would need). It's here. I re-ported the library I mentioned earlier here.

If you aren't happy with that license for that compressor, someone apparently wrote their own here. (It doesn't have a license right this moment, but the author tells me he'll put one up in the next day or two. He mentioned "MIT or whatever" earlier today.) (GPL)

I'm probably going to be writing a tbin loader soon to feature-creep my own project. I'll throw that under MIT as well.

@bjorn

This comment has been minimized.

Show comment
Hide comment
@bjorn

bjorn Jun 9, 2017

Owner

@spacechase0 That's so awesome! Let me know if I can help in any way. Right now, I've got a lot of other things with higher priority, like fixing things for Tiled 1.0.1, mentoring the GSoC students and updating the manual, but I'd really like to work on this as well.

Owner

bjorn commented Jun 9, 2017

@spacechase0 That's so awesome! Let me know if I can help in any way. Right now, I've got a lot of other things with higher priority, like fixing things for Tiled 1.0.1, mentoring the GSoC students and updating the manual, but I'd really like to work on this as well.

@spacechase0

This comment has been minimized.

Show comment
Hide comment
@spacechase0

spacechase0 Jun 16, 2017

Contributor

I got stuck for a while since I didn't know func( a(), b() ) didn't guarantee the execution order of a and b, but my Tbin loader is done now.

Contributor

spacechase0 commented Jun 16, 2017

I got stuck for a while since I didn't know func( a(), b() ) didn't guarantee the execution order of a and b, but my Tbin loader is done now.

@spacechase0

This comment has been minimized.

Show comment
Hide comment
@spacechase0

spacechase0 Jun 20, 2017

Contributor

It looks like I'd just need to create something similar to the Json plugin, right?

I noticed that nothing uses the STL. Should I port the things I already wrote to Qt classes, or is the STL fine? (I assume either way I'll be putting these as a sub-directory in the plugin?)

I see that there is a way to add support for another image format with a QImageIOPlugin. How reasonable would it be to implement it this way? Or would you prefer a more "manual" approach? (If the latter, some guidance would be appreciated. I keep getting lost trying to find things in the codebase.)

Contributor

spacechase0 commented Jun 20, 2017

It looks like I'd just need to create something similar to the Json plugin, right?

I noticed that nothing uses the STL. Should I port the things I already wrote to Qt classes, or is the STL fine? (I assume either way I'll be putting these as a sub-directory in the plugin?)

I see that there is a way to add support for another image format with a QImageIOPlugin. How reasonable would it be to implement it this way? Or would you prefer a more "manual" approach? (If the latter, some guidance would be appreciated. I keep getting lost trying to find things in the codebase.)

@bjorn

This comment has been minimized.

Show comment
Hide comment
@bjorn

bjorn Jun 20, 2017

Owner

It looks like I'd just need to create something similar to the Json plugin, right?

At this point, you could import this .tbin loader and turn it into a Tiled plugin. For that it would need to be stripped from its SFML dependency and the logger statements need to be changed.

Note however that there are some differences between tIDE and Tiled that may be problematic:

  • tIDE appears to support different tile size per layer, Tiled does not. Hopefully SV doesn't use this feature.
  • tIDE appears to store properties for each tile location, which Tiled doesn't directly support (for performance reasons). When loading, objects would need to be created to store these properties and when saving would need to be serialized back to the tile layer.
  • tIDE appears to store full tile animations instead of tile indexes in animated locations, and different frames of an animation can come from different tilesets. In contrast, Tiled defines an animation on the tile and refers to it by tile index as usual, and it can only use frames from the same tileset. This means the loader would need to add new tiles for each unique animation encountered, and animations with frames from different tilesets can't be supported (though care could be taken to keep them intact upon saving by using additional meta-data).

I noticed that nothing uses the STL. Should I port the things I already wrote to Qt classes, or is the STL fine? (I assume either way I'll be putting these as a sub-directory in the plugin?)

Using STL is fine. Actually std::map is used by RangeSet. :-)

Using a subdirectory would be good indeed.

I see that there is a way to add support for another image format with a QImageIOPlugin. How reasonable would it be to implement it this way? Or would you prefer a more "manual" approach? (If the latter, some guidance would be appreciated. I keep getting lost trying to find things in the codebase.)

I'd say we should implement just the .tbin support at first, since it will already help people, and we can then indeed look at adding this.

Owner

bjorn commented Jun 20, 2017

It looks like I'd just need to create something similar to the Json plugin, right?

At this point, you could import this .tbin loader and turn it into a Tiled plugin. For that it would need to be stripped from its SFML dependency and the logger statements need to be changed.

Note however that there are some differences between tIDE and Tiled that may be problematic:

  • tIDE appears to support different tile size per layer, Tiled does not. Hopefully SV doesn't use this feature.
  • tIDE appears to store properties for each tile location, which Tiled doesn't directly support (for performance reasons). When loading, objects would need to be created to store these properties and when saving would need to be serialized back to the tile layer.
  • tIDE appears to store full tile animations instead of tile indexes in animated locations, and different frames of an animation can come from different tilesets. In contrast, Tiled defines an animation on the tile and refers to it by tile index as usual, and it can only use frames from the same tileset. This means the loader would need to add new tiles for each unique animation encountered, and animations with frames from different tilesets can't be supported (though care could be taken to keep them intact upon saving by using additional meta-data).

I noticed that nothing uses the STL. Should I port the things I already wrote to Qt classes, or is the STL fine? (I assume either way I'll be putting these as a sub-directory in the plugin?)

Using STL is fine. Actually std::map is used by RangeSet. :-)

Using a subdirectory would be good indeed.

I see that there is a way to add support for another image format with a QImageIOPlugin. How reasonable would it be to implement it this way? Or would you prefer a more "manual" approach? (If the latter, some guidance would be appreciated. I keep getting lost trying to find things in the codebase.)

I'd say we should implement just the .tbin support at first, since it will already help people, and we can then indeed look at adding this.

@spacechase0

This comment has been minimized.

Show comment
Hide comment
@spacechase0

spacechase0 Jun 20, 2017

Contributor

At this point, you could import this .tbin loader and turn it into a Tiled plugin. For that it would need to be stripped from its SFML dependency and the logger statements need to be changed.

That shouldn't be hard. The SFML stuff is just sf::(Ui|I)nt(8|16|32) which is always (unsigned)? (char|short|int) anyways.

The logger stuff was mainly for my debugging. There used to be a lot more. :P Do you want me to transition those to Tiled or just remove them?

tIDE appears to support different tile size per layer, Tiled does not. Hopefully SV doesn't use this feature.

It doesn't.

tIDE appears to store full tile animations instead of tile indexes in animated locations, ... (though care could be taken to keep them intact upon saving by using additional meta-data).

The game itself always keeps animations on the same tileset, so unless you want "full" tbin support we can ignore that.

Contributor

spacechase0 commented Jun 20, 2017

At this point, you could import this .tbin loader and turn it into a Tiled plugin. For that it would need to be stripped from its SFML dependency and the logger statements need to be changed.

That shouldn't be hard. The SFML stuff is just sf::(Ui|I)nt(8|16|32) which is always (unsigned)? (char|short|int) anyways.

The logger stuff was mainly for my debugging. There used to be a lot more. :P Do you want me to transition those to Tiled or just remove them?

tIDE appears to support different tile size per layer, Tiled does not. Hopefully SV doesn't use this feature.

It doesn't.

tIDE appears to store full tile animations instead of tile indexes in animated locations, ... (though care could be taken to keep them intact upon saving by using additional meta-data).

The game itself always keeps animations on the same tileset, so unless you want "full" tbin support we can ignore that.

@bjorn

This comment has been minimized.

Show comment
Hide comment
@bjorn

bjorn Jun 20, 2017

Owner

The logger stuff was mainly for my debugging. There used to be a lot more. :P Do you want me to transition those to Tiled or just remove them?

For as much as the logged output is actually needed for user feedback about errors, it could be changed to error codes that lead to translatable strings in the Tiled plugin. The parts that are only for debugging should probably just be removed.

Owner

bjorn commented Jun 20, 2017

The logger stuff was mainly for my debugging. There used to be a lot more. :P Do you want me to transition those to Tiled or just remove them?

For as much as the logged output is actually needed for user feedback about errors, it could be changed to error codes that lead to translatable strings in the Tiled plugin. The parts that are only for debugging should probably just be removed.

@spacechase0

This comment has been minimized.

Show comment
Hide comment
@spacechase0

spacechase0 Jun 22, 2017

Contributor

This is probably pretty simple, but I'm getting an error that says "Referenced directory 'G:/Forks/tiled/src/plugins/tbin' does not contain a qbs file." after editing plugins/plugins.qbs. Going to the directory does in fact have a qbs file. Any ideas?

Contributor

spacechase0 commented Jun 22, 2017

This is probably pretty simple, but I'm getting an error that says "Referenced directory 'G:/Forks/tiled/src/plugins/tbin' does not contain a qbs file." after editing plugins/plugins.qbs. Going to the directory does in fact have a qbs file. Any ideas?

@bjorn

This comment has been minimized.

Show comment
Hide comment
@bjorn

bjorn Jun 22, 2017

Owner

The name of the file may need to match the folder name. It is called tbin.qbs?

Owner

bjorn commented Jun 22, 2017

The name of the file may need to match the folder name. It is called tbin.qbs?

@spacechase0

This comment has been minimized.

Show comment
Hide comment
@spacechase0

spacechase0 Jun 22, 2017

Contributor

It does.

I don't know if this is related or a separate issue, but the folder doesn't show in Qt Creator either.

spacechase0@fab57f8

Contributor

spacechase0 commented Jun 22, 2017

It does.

I don't know if this is related or a separate issue, but the folder doesn't show in Qt Creator either.

spacechase0@fab57f8

@bjorn

This comment has been minimized.

Show comment
Hide comment
@bjorn

bjorn Jun 22, 2017

Owner

@spacechase0 I got this when I tried to run qbs:

[bjorn@thor tiled-pr]$ qbs build
Restoring build graph from disk
Resolving project for configuration default
ERROR: /home/bjorn/projects/tiled-pr/src/plugins/tbin/tbin.qbs:8:9 Unexpected token `string literal'

It is due to a missing comma after the first file name in the list.

Btw, please don't hesitate to open a pull request early. It makes it easier for me to help with such problems as well.

Owner

bjorn commented Jun 22, 2017

@spacechase0 I got this when I tried to run qbs:

[bjorn@thor tiled-pr]$ qbs build
Restoring build graph from disk
Resolving project for configuration default
ERROR: /home/bjorn/projects/tiled-pr/src/plugins/tbin/tbin.qbs:8:9 Unexpected token `string literal'

It is due to a missing comma after the first file name in the list.

Btw, please don't hesitate to open a pull request early. It makes it easier for me to help with such problems as well.

bjorn added a commit that referenced this issue Jul 4, 2017

Implement read/write support for the tBIN format (#1622)
The tBIN format is used by the tile map editor tIDE and its rendering
engine xTile. This plugin allows Tiled to read and write tBIN files with
a high level of compatibility. 

The plugin is disabled by default, because the format does not support
all Tiled features. Using it unintentionally could lead to silent data
loss.

Supporting this format is interesting because it is used by Stardew 
Valley (issue #1560).
@bjorn

This comment has been minimized.

Show comment
Hide comment
@bjorn

bjorn Jul 11, 2017

Owner

Thanks to @spacechase0, there should now no longer be a need to go through tIDE to edit Stardew Valley maps. I tested it on some extracted .tbin files and it seemed to work very well! The new plugin is now included in the development snapshot.

@spacechase0 Do you think it makes sense to continue with actually supporting opening maps directly from .xnb files? I guess with your XNB loader it would be no problem to do it for the maps, but it may be hard to make it work for the tileset images (though the plugin could load them in some custom way I think - probably better than doing a Qt image IO plugin). But if in general people anyway need to unpack and repack their data then loading directly from .xnb files isn't that useful.

Owner

bjorn commented Jul 11, 2017

Thanks to @spacechase0, there should now no longer be a need to go through tIDE to edit Stardew Valley maps. I tested it on some extracted .tbin files and it seemed to work very well! The new plugin is now included in the development snapshot.

@spacechase0 Do you think it makes sense to continue with actually supporting opening maps directly from .xnb files? I guess with your XNB loader it would be no problem to do it for the maps, but it may be hard to make it work for the tileset images (though the plugin could load them in some custom way I think - probably better than doing a Qt image IO plugin). But if in general people anyway need to unpack and repack their data then loading directly from .xnb files isn't that useful.

@bjorn

This comment has been minimized.

Show comment
Hide comment
@bjorn

bjorn Jul 12, 2017

Owner

Note to self: don't forget to add the tbin plugin to the Windows MSI installer.

Owner

bjorn commented Jul 12, 2017

Note to self: don't forget to add the tbin plugin to the Windows MSI installer.

@spacechase0

This comment has been minimized.

Show comment
Hide comment
@spacechase0

spacechase0 Jul 16, 2017

Contributor

It could be convenient in some cases, but it definitely is not necessary. People who only edit the map would fine it useful, but those who edit the tilesheets do still need to repack the images. (And those of us who make new maps for SMAPI mods don't need to pack either.)

Contributor

spacechase0 commented Jul 16, 2017

It could be convenient in some cases, but it definitely is not necessary. People who only edit the map would fine it useful, but those who edit the tilesheets do still need to repack the images. (And those of us who make new maps for SMAPI mods don't need to pack either.)

@bjorn bjorn closed this in 1955237 Jul 17, 2017

@bjorn

This comment has been minimized.

Show comment
Hide comment
@bjorn

bjorn Jul 17, 2017

Owner

@spacechase0 Then I'll close this issue. Thanks a lot for your work on this!

Owner

bjorn commented Jul 17, 2017

@spacechase0 Then I'll close this issue. Thanks a lot for your work on this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment