Initial WFSI/WFSSRV implementation (+ some ES additions). #4632

delroth · 2017-01-08T18:08:36Z

This implementation is enough to fully run the Dragon Quest X installer (with tons of stubbing and broken install paths). It also adds support for title installation from ES (which is required for DQX since the game installs itself as a channel, nothing is loaded from disc itself when playing).

There are too many things wrong with the WFS implementation to list here, it should be considered a pre-alpha thing and I'm just thinking of merging it to make merge conflicts less annoying :)

Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_es.cpp

+    File::CreateFullPath(tmd_path);
+
+    File::IOFile fp(tmd_path, "wb");
+    fp.WriteBytes(tmd.data(), tmd.size());


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_usb_wfssrv.cpp

+#include "Core/HW/DVDInterface.h"
+#include "Core/HW/Memmap.h"
+
+u32 num_calls = 0;


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_usb_wfssrv.cpp

+    }
+
+    INFO_LOG(WII_IPC_HLE, "IOCTL_WFS_OPEN(%s, %d) -> %d", path.c_str(), mode, fd);
+    Memory::Write_U16(fd, buffer_out + 0x14);


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.cpp

+void ARCUnpacker::Extract(const WriteCallback& callback)
+{
+  u32 fourcc = Common::swap32(m_whole_file.data());
+  if (fourcc != 0x55AA382D)


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_usb_wfssrv.cpp

+  u64 title_id;
+  DVDInterface::GetVolume().GetTitleID(&title_id);
+
+  Memory::Write_U32(title_id >> 32, command_address + 4);


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_usb_wfssrv.cpp

+
+IPCCommandResult CWII_IPC_HLE_Device_usb_wfssrv::Open(u32 command_address, u32 mode)
+{
+  INFO_LOG(WII_IPC_HLE, "/dev/usb_wfssrv: Open");


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_usb_wfssrv.cpp

+  return GetDefaultReply();
+}
+
+IPCCommandResult CWII_IPC_HLE_Device_usb_wfssrv::Close(u32 command_address, bool force)


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_usb_wfssrv.cpp

+  case IOCTL_WFS_DEVICE_INFO:
+    INFO_LOG(WII_IPC_HLE, "IOCTL_WFS_DEVICE_INFO");
+    Memory::Write_U64(16ull << 30, buffer_out);  // 16GB storage.
+    Memory::Write_U8(4, buffer_out + 8);


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.cpp

+IPCCommandResult CWII_IPC_HLE_Device_wfsi::Open(u32 command_address, u32 mode)
+{
+  INFO_LOG(WII_IPC_HLE, "/dev/wfsi: Open");
+  Memory::Write_U32(GetDeviceID(), command_address + 4);


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.cpp

+{
+  SIOCtlVBuffer command_buffer(command_address);
+  ERROR_LOG(WII_IPC_HLE, "IOCtlV on /dev/usb/wfssrv -- unsupported");
+  Memory::Write_U32(FS_SUCCESS, command_address + 4);


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.cpp

+
+#include "Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.h"
+
+#include <mbedtls/aes.h>


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.cpp

+
+IPCCommandResult CWII_IPC_HLE_Device_wfsi::IOCtlV(u32 command_address)
+{
+  SIOCtlVBuffer command_buffer(command_address);


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.h

+#include "Core/IPC_HLE/WII_IPC_HLE.h"
+#include "Core/IPC_HLE/WII_IPC_HLE_Device.h"
+
+class ARCUnpacker


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.h

+#pragma once
+
+#include <string>
+


JMC47 · 2017-01-08T21:24:56Z

Considering that it only affects Dragon Quest X, I'm of course okay with merging it in an alpha state to make cleanup easier. I'll of course be testing this later today. Congratulations!

JosJuice · 2017-01-08T21:29:21Z

@JMC47 This may affect other things too. For instance, I noticed that trying to copy a channel from an SD card to the NAND using the system menu now gives the message "The data may not have been copied." instead of the system menu claiming that it succeeded. (It never actually succeeds, it just claims to succeed.)

JMC47 · 2017-01-08T21:41:25Z

That's still fine, it's all fringe things.

JosJuice · 2017-01-08T21:49:38Z

Yeah, the added ES function isn't exactly something that's used by emulated code that's working correctly in master.

Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_usb_wfssrv.cpp

+  u64 title_id;
+  DVDInterface::GetVolume().GetTitleID(&title_id);
+
+  Memory::Write_U32(title_id >> 32, command_address + 4);


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.cpp

+
+#include "Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.h"
+
+#include <mbedtls/aes.h>


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.cpp

+IPCCommandResult CWII_IPC_HLE_Device_wfsi::IOCtlV(u32 command_address)
+{
+  ERROR_LOG(WII_IPC_HLE, "IOCtlV on /dev/wfsi -- unsupported");
+  Memory::Write_U32(IPC_SUCCESS, command_address + 4);


WFSI calls into ES to perform this operation, so expose a way for us to do the same thing.

Refactor the existing DiscIO::AddTicket to not require the caller to pass the requested title ID. We do not have the title ID in the ES case, and it needs to be extracted from the ticket. Since this is always a safe operation to do (title ID is always in the ticket), the implementation is made default.

JMC47 · 2017-01-14T15:03:00Z

Just note: this Pull Request has been verified to add enough functionality for the Mario Kart Wii and Wii Fit Channels to be successfully installed.

Wii Fit Channel runs as expected, I haven't been able to run the MKWii Channel yet due to unrelated issues.

Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_usb_wfssrv.cpp

+{
+  SIOCtlVBuffer command_buffer(command_address);
+  ERROR_LOG(WII_IPC_HLE, "IOCtlV on /dev/usb/wfssrv -- unsupported");
+  Memory::Write_U32(FS_EINVAL, command_address + 4);


Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_es.cpp

+    std::vector<u8> ticket(Buffer.InBuffer[0].m_Size);
+    Memory::CopyFromEmu(ticket.data(), Buffer.InBuffer[0].m_Address, Buffer.InBuffer[0].m_Size);
+    DiscIO::AddTicket(ticket);
+    break;


Source/Core/DiscIO/NANDContentLoader.cpp

-  }
-  else
+  std::vector<u8> ticket = SignedTicketToTicket(signed_ticket);
+  if (ticket.size() == 0)


Source/Core/DiscIO/NANDContentLoader.cpp

+std::vector<u8> FindTicket(u64 title_id)
+{
+  std::vector<u8> signed_ticket = FindSignedTicket(title_id);
+  if (!signed_ticket.size())


These two functions load either a signed ticket or a raw ticket from the emulated NAND. The ticket signature skip is refactored out of the ticket writing in order to be usable by the raw ticket reading function.

This function has more uses than just in DiscIO (e.g. WFS).

This library implements basic parsing support for some of the IOS ES formats we need to extract data from. Currently only implements TMD functions, but some ticket handling functions from DiscIO should likely be moved here in the future.

There are several things wrong with this implementation. The first being that since we still don't have a proper ticket/tmd handling library, we hardcode offsets once again to fetch TMD fields. The second being that we don't stream data to the disk and we buffer everything in memory. The third being that we don't properly fetch the content index for decryption, which is prone to breaking. But hey, it works well enough to install the DQX channel!

Defaults to $USERDIR/WFS. Used to store the contents normally stored on WFS mass storage devices.

Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.h

+  std::string m_device_name;
+
+  mbedtls_aes_context m_aes_ctx;
+  u8 m_aes_key[0x10];


The current implementations do many things wrong but work well enough to run the Dragon Quest X installer until the very end. The game itself crashes when being launched from its System Menu channel unfortunately so it is hard to verify whether the install properly worked or not. There are plenty of "TODO(wfs)" sprinkled around this codebase with things that are knowingly done wrong. The most important one right now is that content extraction is done by buffering everything into memory instead of properly streaming the data to disk (and processing asynchronously), which causes freezes. It is likely to not cause any practical issues since only the installer and the updater should use this anyway.

delroth force-pushed the wfs branch 2 times, most recently from f83a923 to b1221a9 Compare January 8, 2017 18:26

iwubcode reviewed Jan 8, 2017

View reviewed changes

Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_es.cpp

File::CreateFullPath(tmd_path);

File::IOFile fp(tmd_path, "wb");

fp.WriteBytes(tmd.data(), tmd.size());

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

Warepire reviewed Jan 8, 2017

View reviewed changes

Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_usb_wfssrv.cpp

#include "Core/HW/DVDInterface.h"

#include "Core/HW/Memmap.h"

u32 num_calls = 0;

This comment was marked as off-topic.

Sign in to view

This comment was marked as off-topic.

Sign in to view

delroth force-pushed the wfs branch from b1221a9 to 24d33cf Compare January 8, 2017 18:42

iwubcode reviewed Jan 8, 2017

View reviewed changes

leoetlino reviewed Jan 8, 2017

View reviewed changes

delroth force-pushed the wfs branch 2 times, most recently from 9e7513f to 89e97fd Compare January 9, 2017 02:28

leoetlino reviewed Jan 9, 2017

View reviewed changes

delroth force-pushed the wfs branch from 89e97fd to f12ccf4 Compare January 14, 2017 14:21

delroth added 2 commits January 14, 2017 15:23

IOS/ES: Factor out the ES_Decrypt implementation.

c30635c

WFSI calls into ES to perform this operation, so expose a way for us to do the same thing.

delroth force-pushed the wfs branch 2 times, most recently from b81d424 to c9c48de Compare January 14, 2017 14:33

delroth force-pushed the wfs branch 2 times, most recently from c260742 to c3057f8 Compare January 14, 2017 15:10

leoetlino reviewed Jan 14, 2017

View reviewed changes

delroth force-pushed the wfs branch from c3057f8 to e957df7 Compare January 14, 2017 15:18

leoetlino reviewed Jan 14, 2017

View reviewed changes

delroth force-pushed the wfs branch from e957df7 to 23fabdd Compare January 14, 2017 16:04

delroth added 3 commits January 14, 2017 17:06

DiscIO: Implement functions to lookup tickets

650a1fd

These two functions load either a signed ticket or a raw ticket from the emulated NAND. The ticket signature skip is refactored out of the ticket writing in order to be usable by the raw ticket reading function.

DiscIO: Export GetKeyFromTicket

334ddf7

This function has more uses than just in DiscIO (e.g. WFS).

IOS: Add ESFormats.{cpp,h}.

0a5cfd8

This library implements basic parsing support for some of the IOS ES formats we need to extract data from. Currently only implements TMD functions, but some ticket handling functions from DiscIO should likely be moved here in the future.

delroth added 2 commits January 14, 2017 17:06

CommonPaths: Add D_WFSROOT_IDX.

45d27f7

Defaults to $USERDIR/WFS. Used to store the contents normally stored on WFS mass storage devices.

delroth force-pushed the wfs branch from 23fabdd to 3376b0e Compare January 14, 2017 16:06

lioncash reviewed Jan 14, 2017

View reviewed changes

Source/Core/Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.h

std::string m_device_name;

mbedtls_aes_context m_aes_ctx;

u8 m_aes_key[0x10];

This comment was marked as off-topic.

Sign in to view

delroth force-pushed the wfs branch from 3376b0e to 45a1232 Compare January 14, 2017 18:56

delroth merged commit be6e7d7 into dolphin-emu:master Jan 14, 2017


		#include "Core/IPC_HLE/WII_IPC_HLE_Device_wfsi.h"

		#include <mbedtls/aes.h>

Initial WFSI/WFSSRV implementation (+ some ES additions). #4632

Initial WFSI/WFSSRV implementation (+ some ES additions). #4632

Conversation

delroth commented Jan 8, 2017 • edited Loading

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

JMC47 commented Jan 8, 2017

JosJuice commented Jan 8, 2017

JMC47 commented Jan 8, 2017

JosJuice commented Jan 8, 2017

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

JMC47 commented Jan 14, 2017 • edited Loading

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

This comment was marked as off-topic.

delroth commented Jan 8, 2017 •

edited

Loading

JMC47 commented Jan 14, 2017 •

edited

Loading