[single-exe] Run from bundle on Windows (#26904)

* [single-exe] Run from bundle on Windows

This commit has two main changes:
* Add a simplified `corebundle` host on Windows, similar to `unixcorebundle`
* Enable assemblies embedded within single-exe bundles to be loaded directly from the bundle.

The CoreCLR native binaries are not linked into the corebundle host yet, so there need to be located separately on disk beside the single-exe app.

`corebundle` host:
Add a new CoreBundle host – the prototype host for single-file self-contained apps on Windows
This code is adapted from coreconsole, with a few changes:
* This host uses the coreclr_initialize() interface similar to AppHost, unlike other Windows hosts in `coreclr\src\hosts`
* The path computations are done in UTF8 (similar to unixcorebundle) and not Unicode
* Calls bundle-handling logic, and sets up appropriate runtime properties.

`app_bundle` abstraction:
* Added the `app_bundle_t` class to represent the bundle for the currently executing application
* This class is the bundle-processing module's interface with the outside world, and shares code common to `corebundle` and `unixcorebundle`

Bundle handling code:
* Move `bundle` directory to `coreclr/src/hosts` so that it can be shared by `corebundle` and `unixcorebundle`
* Remove dummy tracing code and trace files
* Add Windows PAL definitions so that bundle handling works on the Windows host
* Remove some unnecessary PAL definitions

Loading the contents of the bundle:
* IL assemblies are loaded directly from the bundle using `FlatImageLayout`
* R2R assemblies are loaded through `ConvertedImageLayout` which involves copying sections to appropriate offsets.

Limitations:

There are a few shortcomings to running single-exe apps on Windows, as compared to Linux.
The Windows mapping routines have the following limitations:
* [CreateFileMapping](https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-createfilemappinga) has no option to create a mapping for a part of the file (no offset argument).
    This means that:
    * We cannot use the (SEC_IMAGE) attribute to perform automatic section-wise loading (circumventinc alignment requirements) of bundled assemblies directly.
    * Instead we need to map each section independently.
* [MapViewOfFile]( https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-mapviewoffileex) can only map parts of a file aligned at [memory allocation granularity]( https://docs.microsoft.com/en-us/windows/win32/api/sysinfoapi/ns-sysinfoapi-system_info), which is 64KB.
    This means that:
    * Each section within the assemblies should be aligned at 64KB – which is not guaranteed by the crossgen compiler.
        * We therefore map the entire assembly and perform an in-memory copy of the sections to appropriate offsets.
        * This behavior is achieved by using `ConvertedImageLayout`
    * In order to memory-map one embedded assembly at a time, the assemblies in the bundle must be aligned at 64KB boundaries.
        * A prototype bundler with this behavior for testing purposes is available in this branch: https://github.com/swaroop-sridhar/core-setup/tree/single-exe
        * This change is strictly not necessary while copying out individual sections after memory-map (because we can align-down and map)
        * But this change is closer to the final no-copy solution.

In the long term, the solution to the mapping problem would involve considerations such as:
* Use the new crossgen AOT compiler to compile all assemblies in a version bubble into one PE assembly, with a few aligned sections.
* Create the big-assembly into the host with proper alignment, so that the single-exe bundle can be loaded without copies at run time.

* Fix a build error in the lab.

src/coreclr/hosts/CMakeLists.txt

@@ -3,6 +3,7 @@ include_directories(inc)
 if(WIN32)
   add_subdirectory(corerun)
   add_subdirectory(coreconsole)
+  add_subdirectory(corebundle)
   add_subdirectory(coreshim)
 else(WIN32)
   include_directories(${CMAKE_CURRENT_BINARY_DIR})

src/coreclr/hosts/bundle/CMakeLists.txt

@@ -0,0 +1,34 @@
+set(Bundle_Common_SOURCES  
+    ${CMAKE_CURRENT_LIST_DIR}/app_bundle.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/file_entry.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/header.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/manifest.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/marker.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/reader.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/runner.cpp
+    ${CMAKE_CURRENT_LIST_DIR}/utils.cpp
+)
+
+set(Bundle_Windows_SOURCES  
+    ${Bundle_Common_SOURCES}
+    ${CMAKE_CURRENT_LIST_DIR}/pal_windows.cpp
+)
+
+set(Bundle_Unix_SOURCES  
+    ${Bundle_Common_SOURCES}
+    ${CMAKE_CURRENT_LIST_DIR}/pal_unix.cpp
+)
+
+set(Bundle_HEADERS
+    ${CMAKE_CURRENT_LIST_DIR}/app_bundle.h
+    ${CMAKE_CURRENT_LIST_DIR}/error_codes.h
+    ${CMAKE_CURRENT_LIST_DIR}/file_entry.h
+    ${CMAKE_CURRENT_LIST_DIR}/file_type.h
+    ${CMAKE_CURRENT_LIST_DIR}/header.h
+    ${CMAKE_CURRENT_LIST_DIR}/manifest.h
+    ${CMAKE_CURRENT_LIST_DIR}/marker.h
+    ${CMAKE_CURRENT_LIST_DIR}/pal.h
+    ${CMAKE_CURRENT_LIST_DIR}/reader.h
+    ${CMAKE_CURRENT_LIST_DIR}/runner.h
+    ${CMAKE_CURRENT_LIST_DIR}/utils.h
+)

src/coreclr/hosts/bundle/app_bundle.cpp

@@ -0,0 +1,34 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+
+#include "app_bundle.h"
+#include "marker.h"
+#include "runner.h"
+
+using namespace bundle;
+
+runner_t* app_bundle_t::s_runner = nullptr;
+
+bool app_bundle_t::init(const char* exe_path)
+{
+    if (!bundle::marker_t::is_bundle())
+    {
+        return false;
+    }
+
+    pal::string_t self_path;
+    pal::clr_palstring(exe_path, self_path);
+
+    static runner_t runner(exe_path);
+    s_runner = &runner;
+
+    StatusCode status = runner.process();
+
+    return status == StatusCode::Success;
+}
+
+bool app_bundle_t::probe(const char* path, int64_t* size, int64_t* offset)
+{
+    return s_runner->probe(path, size, offset);
+}

src/coreclr/hosts/bundle/app_bundle.h

@@ -0,0 +1,28 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+
+// This class represents the bundle for the currently executing application
+// It is the bundle-processing module's interface with the outside world.
+
+#ifndef __APP_BUNDLE_H__
+#define __APP_BUNDLE_H__
+
+#include <stdint.h>
+
+namespace bundle
+{
+    class runner_t;
+
+    class app_bundle_t
+    {
+    public:
+        static bool init(const char *path);
+        static bool probe(const char* path, int64_t* size, int64_t* offset);
+
+    private:
+        static runner_t *s_runner;
+    };
+}
+
+#endif // __APP_BUNDLE_H__

src/coreclr/hosts/unixcorebundle/bundle/file_entry.cpp → src/coreclr/hosts/bundle/file_entry.cpp

@@ -3,7 +3,6 @@
 // See the LICENSE file in the project root for more information.
 
 #include "file_entry.h"
-#include "trace.h"
 #include "error_codes.h"
 
 using namespace bundle;
@@ -15,7 +14,7 @@ bool file_entry_t::is_valid() const
 }
 
 // Fixup a path to have current platform's directory separator.
-void fixup_path_separator(pal::string_t& path)
+void fixup_dir_separator(pal::string_t& path)
 {
     const pal::char_t bundle_dir_separator = '/';
 
@@ -38,13 +37,11 @@ file_entry_t file_entry_t::read(reader_t &reader)
 
     if (!entry.is_valid())
     {
-        trace::error(_X("Failure processing application bundle; possible file corruption."));
-        trace::error(_X("Invalid FileEntry detected."));
         throw StatusCode::BundleExtractionFailure;
     }
 
     reader.read_path_string(entry.m_relative_path);
-    fixup_path_separator(entry.m_relative_path);
+    fixup_dir_separator(entry.m_relative_path);
 
     return entry;
 }

src/coreclr/hosts/unixcorebundle/bundle/header.cpp → src/coreclr/hosts/bundle/header.cpp

@@ -5,7 +5,6 @@
 #include "header.h"
 #include "reader.h"
 #include "error_codes.h"
-#include "trace.h"
 
 using namespace bundle;
 
@@ -22,9 +21,6 @@ header_t header_t::read(reader_t& reader)
 
     if (!fixed_header->is_valid())
     {
-        trace::error(_X("Failure processing application bundle."));
-        trace::error(_X("Bundle header version compatibility check failed."));
-
         throw StatusCode::BundleExtractionFailure;
     }
 

src/coreclr/hosts/bundle/pal.h

@@ -0,0 +1,68 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+
+#ifndef PAL_H
+#define PAL_H
+
+#include <string>
+#include <vector>
+#include <fstream>
+#include <sstream>
+#include <iostream>
+#include <cstring>
+#include <cstdarg>
+#include <cstdint>
+#include <cassert>
+
+#if defined(_WIN32)
+#include <windows.h>
+#else // defined(_WIN32)
+#include <cstdlib>
+#include <unistd.h>
+#include <libgen.h>
+#include <mutex>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/mman.h>
+#endif // defined(_WIN32)
+
+#if defined(_WIN32)
+#define DIR_SEPARATOR '\\'
+#define PATH_MAX MAX_PATH
+
+#else // defined(_WIN32)
+#define DIR_SEPARATOR '/'
+
+#if !defined(PATH_MAX) 
+#define PATH_MAX    4096
+#endif
+#endif // defined(_WIN32)
+
+namespace pal
+{
+    typedef std::basic_ifstream<char> ifstream_t;
+    typedef std::istreambuf_iterator<ifstream_t::char_type> istreambuf_iterator_t;
+    typedef std::basic_istream<char> istream_t;
+
+    typedef char char_t;
+    typedef std::string string_t;
+    typedef std::stringstream stringstream_t;
+
+    inline bool clr_palstring(const char* cstr, string_t& out) { out.assign(cstr); return true; }
+
+#if defined(_WIN32)
+    typedef HMODULE dll_t;
+    typedef FARPROC proc_t;
+    inline int pathcmp(const char_t* path1, const char_t* path2) { return ::_stricmp(path1, path2); }
+#else // defined(_WIN32)
+    typedef void* dll_t;
+    typedef void* proc_t;
+    inline int pathcmp(const char_t* path1, const char_t* path2) { return ::strcmp(path1, path2); }
+#endif // defined(_WIN32)
+
+    void* map_file_readonly(const string_t& path, size_t& length);
+    bool unmap_file(void* addr, size_t length);
+}
+
+#endif // PAL_H

src/coreclr/hosts/bundle/pal_unix.cpp

@@ -0,0 +1,42 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+
+#include "pal.h"
+
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <dirent.h>
+
+void* pal::map_file_readonly(const pal::string_t& path, size_t& length)
+{
+    int fd = open(path.c_str(), O_RDONLY, (S_IRUSR | S_IRGRP | S_IROTH));
+    if (fd == -1)
+    {
+        return nullptr;
+    }
+
+    struct stat buf;
+    if (fstat(fd, &buf) != 0)
+    {
+        close(fd);
+        return nullptr;
+    }
+
+     length = buf.st_size;
+    void* address = mmap(nullptr, length, PROT_READ, MAP_SHARED, fd, 0);
+
+     if(address == nullptr)
+    {
+        close(fd);
+        return nullptr;
+    }
+
+     close(fd);
+    return address;
+}
+
+bool pal::unmap_file(void* addr, size_t length) 
+{ 
+    return munmap(addr, length) == 0; 
+}

src/coreclr/hosts/bundle/pal_windows.cpp

@@ -0,0 +1,46 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+// See the LICENSE file in the project root for more information.
+#include "pal.h"
+
+void* pal::map_file_readonly(const pal::string_t& path, size_t& length)
+{
+    HANDLE file = CreateFileA(path.c_str(), GENERIC_READ, FILE_SHARE_READ, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
+
+    if (file == INVALID_HANDLE_VALUE)
+    {
+        return nullptr;
+    }
+
+    LARGE_INTEGER fileSize;
+    if (GetFileSizeEx(file, &fileSize) == 0)
+    {
+        CloseHandle(file);
+        return nullptr;
+    }
+    length = (size_t)fileSize.QuadPart;
+
+    HANDLE map = CreateFileMappingW(file, NULL, PAGE_READONLY, 0, 0, NULL);
+
+    if (map == NULL)
+    {
+        CloseHandle(file);
+        return nullptr;
+    }
+
+    void* address = MapViewOfFile(map, FILE_MAP_READ, 0, 0, 0);
+
+    if (map == NULL)
+    {
+        CloseHandle(file);
+        return nullptr;
+    }
+
+    return address;
+}
+
+bool pal::unmap_file(void* addr, size_t length) 
+{ 
+    return UnmapViewOfFile(addr) != 0; 
+}
+

src/coreclr/hosts/unixcorebundle/bundle/reader.cpp → src/coreclr/hosts/bundle/reader.cpp

@@ -4,7 +4,6 @@
 
 #include "reader.h"
 #include "error_codes.h"
-#include "trace.h"
 #include <memory>
 
 using namespace bundle;
@@ -17,8 +16,6 @@ const int8_t* reader_t::add_without_overflow(const int8_t* ptr, int64_t len)
     // even if the actual arthmetic didn't overflow.
     if (new_ptr < ptr)
     {
-        trace::error(_X("Failure processing application bundle; possible file corruption."));
-        trace::error(_X("Arithmetic overflow computing bundle-bounds."));
         throw StatusCode::BundleExtractionFailure;
     }
 
@@ -29,8 +26,6 @@ void reader_t::set_offset(int64_t offset)
 {
     if (offset < 0 || offset >= m_bound)
     {
-        trace::error(_X("Failure processing application bundle; possible file corruption."));
-        trace::error(_X("Arithmetic overflow while reading bundle."));
         throw StatusCode::BundleExtractionFailure;
     }
 
@@ -44,8 +39,6 @@ void reader_t::bounds_check(int64_t len)
     // It is legal for post_read_ptr == m_bound_ptr after reading the last byte.
     if (m_ptr < m_base_ptr || post_read_ptr > m_bound_ptr)
     {
-        trace::error(_X("Failure processing application bundle; possible file corruption."));
-        trace::error(_X("Bounds check failed while reading the bundle."));
         throw StatusCode::BundleExtractionFailure;
     }
 }
@@ -70,9 +63,6 @@ size_t reader_t::read_path_length()
         if (second_byte & 0x80)
         {
             // There can be no more than two bytes in path_length
-            trace::error(_X("Failure processing application bundle; possible file corruption."));
-            trace::error(_X("Path length encoding read beyond two bytes."));
-
             throw StatusCode::BundleExtractionFailure;
         }
 
@@ -81,8 +71,6 @@ size_t reader_t::read_path_length()
 
     if (length <= 0 || length > PATH_MAX)
     {
-        trace::error(_X("Failure processing application bundle; possible file corruption."));
-        trace::error(_X("Path length is zero or too long."));
         throw StatusCode::BundleExtractionFailure;
     }
 
@@ -95,5 +83,5 @@ void reader_t::read_path_string(pal::string_t &str)
     std::unique_ptr<uint8_t[]> buffer{ new uint8_t[size + 1] };
     read(buffer.get(), size);
     buffer[size] = 0; // null-terminator
-    pal::clr_palstring(reinterpret_cast<const char*>(buffer.get()), &str);
+    pal::clr_palstring(reinterpret_cast<const char*>(buffer.get()), str);
 }

src/coreclr/hosts/unixcorebundle/bundle/reader.h → src/coreclr/hosts/bundle/reader.h

@@ -40,7 +40,7 @@ namespace bundle
         void read(void* dest, int64_t len)
         {
             bounds_check(len);
-            memcpy(dest, m_ptr, len);
+            memcpy(dest, m_ptr, (size_t)len);
             m_ptr += len;
         }