Skip to content

Conversation

@elkaamee326
Copy link
Contributor

@elkaamee326 elkaamee326 commented Apr 18, 2021

This build of NASM was confirmed (unofficially) that it resolved the error, by loading a large byte array in the Userkit. If you are currently using the User Kit, download the build of NASM to C:\Users\YourUserName\.nuget\packages\cosmos.build\0.1.0-localbuild{YourUserKitVersionOfCosmos.Build}\tools\nasm\win. It should also work with the dev kit as well. (Both builds should say the Cosmos.Build package version in your local directory).

Note: * replace {YourUserKitVersionOfCosmos.Build} with the nuget build version of Cosmos.Build...

The build was built from an open source 2012 version of NASM's source code. The cause of the error was #define DEADMAN_LIMIT. The evident solution was to increase the value and then rebuild NASM.

This build of Nasm resolved the tedious error, which is known as an Interminable Macro Recursion. For verification, it was capable of loading a larger byte array without displaying the error. Now, you can add more code to your project without it displaying an error. Just one thing to note: if the asset (or array) that you are importing is too large, VS will start to slow down because of the amount of RAM usage. Also, if (the asset) is way too large VS might restart itself.
@elkaamee326
Copy link
Contributor Author

elkaamee326 commented Apr 18, 2021

Would there be anything else that is required, in order to submit this pull request?

@charlesbetros
Copy link
Member

Is this an official build of nasm? If so, what version and is there an official link to download it? I don't feel comfortable merging an executable if we don't know the contents or where the build came from.

@elkaamee326
Copy link
Contributor Author

elkaamee326 commented Apr 19, 2021

The build was unofficial. So, instead of importing that unofficial Nasm build, I will erase this build from the pull request and research online to see and test if a modern Nasm official build will get rid of this error. Also, I will send the build link of the official build and the source code if needed. So, I will be changing this pull request to Update Nasm Build Version.

@elkaamee326 elkaamee326 changed the title Fixed Nasm Interminable Macro Recursion error... Update Nasm Build version... Apr 19, 2021
@elkaamee326
Copy link
Contributor Author

Now, all that is left is testing...

@elkaamee326
Copy link
Contributor Author

Also, this is where I downloaded the official build: https://www.nasm.us/pub/nasm/releasebuilds/2.15.05/win32/

@quajak
Copy link
Member

quajak commented Apr 19, 2021

We had talked about switching to FASM at some point, which I think also fixes the infinite recursion, while also providing a speed up. @elkaamee326 could you compare the new nasm version with fasm?

@elkaamee326
Copy link
Contributor Author

I will document the fasm and nasm build times and see if it resolves the interminable macro recursion error.

@elkaamee326
Copy link
Contributor Author

elkaamee326 commented Apr 19, 2021

If we use FASM, then we will have to change Cosmos.Build.Targets. I am not sure exactly how to make it work. When I tried to do it, it invoked fasm, but it then said FASM exited with the code 0. It was not an interminable macro recursion error, it is a Cosmos.Build.Tasks error. Some help with changing this would be apprectiated. Also, the new official NASM build resolved the interminable Macro recursion error with a build time of 50 secs. I will do a build with FASM (that is not integrated with Cosmos) and see if the build times are the same. New: for FASM we would need to update il2cpu, because FASM reports that there is an illegal statement at line 1: #ifndef ELF_COMPILATION. To make this work, it would require putting effort in changing il2cpu and fixing the build targets. Having the interminable Macro recursion error completely fixed is sufficient.

@quajak
Copy link
Member

quajak commented Apr 19, 2021

Thanks for looking into it. I must have misremebered since I thought we could just replace the NASM exe with FASM. Sorry about that.

@quajak
Copy link
Member

quajak commented Apr 21, 2021

It turns out I was thinking of yasm, not fasm. https://yasm.tortall.net/ Could you please compare the new nasm with yasm?

@elkaamee326
Copy link
Contributor Author

elkaamee326 commented Apr 21, 2021

Any assembler should work, as long as the command line params are the same for Nasm and as long as most things are similar, I will give it a shot. :)

@charlesbetros
Copy link
Member

charlesbetros commented Apr 21, 2021 via email

@quajak
Copy link
Member

quajak commented Apr 22, 2021

Since yasm states Yasm is a complete rewrite of the NASM assembler that shouldnt be the problem. My main hope is to get the compile time further down while removing that limit on the length of inline arrays.

@elkaamee326
Copy link
Contributor Author

elkaamee326 commented Apr 23, 2021

Just to make sure that I am using the right args for Yasm, what is the default Nasm arg that is used without going into the Project Properties settings? This is necessary, to ensure the correct compilation using Yasm.

@quajak
Copy link
Member

quajak commented Apr 24, 2021

The code for executing NASM is here https://github.com/CosmosOS/Cosmos/blob/master/source/Cosmos.Build.Tasks/Nasm.cs GenerateCommandLineCommands sets all the command line arguments. I thought I had tried YASM once, where I just renamed it to nasm.exe and Cosmos was fine using it.

@elkaamee326
Copy link
Contributor Author

Apparently, for the new version of Nasm, it didn't completely resolve the interminable macro recursion error. I will do a yasm test to confirm it resolves the error.

@quajak
Copy link
Member

quajak commented Apr 25, 2021

Apparently, for the new version of Nasm, it didn't completely resolve the interminable macro recursion error. I will do a yasm test to confirm it resolves the error.

What does the new version do? Does it allow larger byte arrays or is there another limitation?

@elkaamee326 elkaamee326 changed the title Update Nasm Build version... [WIP] Update Nasm Build version... May 9, 2021
@elkaamee326
Copy link
Contributor Author

I just figured out the cause of those two errors. I compiled a program as elf for the new Nasm and Yasm compared and for some reason, both of the assemblers don't support the [Map] directive. This is because the developers for the assemblers removed support for elf compilation with the map directive. Hence the error. Would this mean we would need to go into IL2CPU.cs? Or even modify IL2CPU so that it doesn't show this error. Do you recommend removing elf support and on project startup, should we set the build configuration to .bin by default? Personally, I don't know very much about IL2CPU and how we can change it to remove the [map] directive, but I know that it will require a lot of work.

@quajak
Copy link
Member

quajak commented May 20, 2021

Good find! Looking at the emitted asm file,

%ifndef ELF_COMPILATION
use32
org 0x1000000
[map all main.map]
%endif

does that mean our %ifndef ELF_COMPILATION check is incorrect?
I assume that the correct fix would be to stop emitting the map directive when compiling for elf which most likely has to be done in IL2CPU. We are currently emitting the map directive at https://github.com/CosmosOS/IL2CPU/blob/master/source/Cosmos.IL2CPU/CosmosAssembler.cs#L549.

@elkaamee326
Copy link
Contributor Author

elkaamee326 commented May 20, 2021

No, its just that Elf doesn't support the map directive (as well as mapping) anymore because now only .bin supports it I believe. Also, in the new versions of Yasm and Nasm, they both don't support the org directive when compiling with elf. I think we should either remove that statement:
%ifndef ELF_COMPILATION use32 org 0x1000000 [map all main.map] %endif
or, we can just remove all of the ELF_COMPILATION statements.

@elkaamee326
Copy link
Contributor Author

elkaamee326 commented May 20, 2021

Okay, I got elf to work with Yasm and IL2CPU, but now it displays this when it tries to extract a map file (my build log):

1>------ Build started: Project: CosmosKernel1, Configuration: Debug Any CPU ------
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\Apps\Paint.cs(13,13,13,14): warning CS0108: 'Paint.x' hides inherited member 'App.x'. Use the new keyword if hiding was intended.
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\Apps\Paint.cs(13,16,13,17): warning CS0108: 'Paint.y' hides inherited member 'App.y'. Use the new keyword if hiding was intended.
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\Apps\Paint.cs(12,13,12,23): warning CS0649: Field 'Paint.PaintBrush' is never assigned to, and will always have its default value null
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\LogonUI\LoginManager.cs(22,28,22,37): warning CS0649: Field 'LoginManager.TextBox_X' is never assigned to, and will always have its default value 0
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\LogonUI\LoginManager.cs(22,39,22,48): warning CS0649: Field 'LoginManager.TextBox_Y' is never assigned to, and will always have its default value 0
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\LogonUI\LoginManager.cs(23,28,23,41): warning CS0169: The field 'LoginManager.LoginUIFont_X' is never used
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\LogonUI\LoginManager.cs(21,35,21,40): warning CS0169: The field 'LoginManager.LogoY' is never used
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\Dock\Dock.cs(20,14,20,16): warning CS0169: The field 'Dock.AM' is never used
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\Dock\Dock.cs(20,18,20,20): warning CS0169: The field 'Dock.PM' is never used
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\Apps\Windows\Controls\Textbox.cs(113,22,113,29): warning CS0649: Field 'Textbox.cLength' is never assigned to, and will always have its default value 0
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\LogonUI\LoginManager.cs(21,28,21,33): warning CS0169: The field 'LoginManager.LogoX' is never used
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\LogonUI\LoginManager.cs(23,43,23,56): warning CS0169: The field 'LoginManager.LoginUIFont_Y' is never used
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\Apps\Windows\Controls\Textbox.cs(12,18,12,26): warning CS0169: The field 'Textbox.keyEvent' is never used
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\Image Support\Bitmap\Bitmap.cs(16,18,16,27): warning CS0169: The field 'Bitmap.mDebugger' is never used
1>C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\GUI\Apps\Windows\App.cs(37,14,37,21): warning CS0414: The field 'App.hovered' is assigned but its value is never used
1>CosmosKernel1 -> C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\bin\Debug\netcoreapp2.0\cosmos\CosmosKernel1.dll
1>Message: Executing IL2CPU on assembly
1>Message: Kernel Base: Cosmos.System.Kernel
1>Message: Checking target assembly: C:\Users\elkaa\Documents\CosmosKernel1\CosmosKernel1\bin\Debug\netcoreapp2.0\cosmos\CosmosKernel1.dll
1>Detecting fields for type 'IL2CPU.Debug.Symbols.FIELD_INFO'
1>Detecting fields for type 'IL2CPU.Debug.Symbols.FIELD_MAPPING'
1>Detecting fields for type 'IL2CPU.Debug.Symbols.MethodIlOp'
1>Detecting fields for type 'IL2CPU.Debug.Symbols.Document'
1>Detecting fields for type 'IL2CPU.Debug.Symbols.AssemblyFile'
1>Detecting fields for type 'IL2CPU.Debug.Symbols.Method'
1>Detecting fields for type 'IL2CPU.Debug.Symbols.LOCAL_ARGUMENT_INFO'
1>IL2CPU task took 00:01:22.7998446
1>Nasm task took 00:01:31.2390425           
                        //-------  (manually added) This is a pretty big project. On a smaller project, Yasm compiled the code in only 5secs... ------ ////
1>-Ttext 0x2000000 -Tdata 0x1000000 -e Kernel_Start -o bin\Debug\netcoreapp2.0\cosmos\CosmosKernel1.bin bin\Debug\netcoreapp2.0\cosmos\CosmosKernel1.obj
1>LD task took 00:00:01.6896300
1>Extracting Map file...
1>Extracting Map file took 00:00:28.4942017
1>C:\Users\elkaa\.nuget\packages\cosmos.build\0.1.0-localbuild20210416035217\build\Cosmos.Build.targets(258,9): error MSB4018: The "ExtractMapFromElfFile" task failed unexpectedly.
1>C:\Users\elkaa\.nuget\packages\cosmos.build\0.1.0-localbuild20210416035217\build\Cosmos.Build.targets(258,9): error MSB4018: Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 19: 'UNIQUE constraint failed: Labels.Name'.
1>C:\Users\elkaa\.nuget\packages\cosmos.build\0.1.0-localbuild20210416035217\build\Cosmos.Build.targets(258,9): error MSB4018:    at Microsoft.Data.Sqlite.SqliteException.ThrowExceptionForRC(Int32 rc, sqlite3 db)
1>C:\Users\elkaa\.nuget\packages\cosmos.build\0.1.0-localbuild20210416035217\build\Cosmos.Build.targets(258,9): error MSB4018:    at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader(CommandBehavior behavior)
1>C:\Users\elkaa\.nuget\packages\cosmos.build\0.1.0-localbuild20210416035217\build\Cosmos.Build.targets(258,9): error MSB4018:    at Microsoft.Data.Sqlite.SqliteCommand.ExecuteReader()
1>C:\Users\elkaa\.nuget\packages\cosmos.build\0.1.0-localbuild20210416035217\build\Cosmos.Build.targets(258,9): error MSB4018:    at Microsoft.Data.Sqlite.SqliteCommand.ExecuteNonQuery()
1>C:\Users\elkaa\.nuget\packages\cosmos.build\0.1.0-localbuild20210416035217\build\Cosmos.Build.targets(258,9): error MSB4018:    at IL2CPU.Debug.Symbols.SQL.Exec(String aSql) in C:\Users\elkaa\Desktop\Source\IL2CPU\source\IL2CPU.Debug.Symbols\SQL.cs:line 27
1>C:\Users\elkaa\.nuget\packages\cosmos.build\0.1.0-localbuild20210416035217\build\Cosmos.Build.targets(258,9): error MSB4018:    at IL2CPU.Debug.Symbols.SQL.MakeIndex(String aTable, String aCol, Boolean aUnique) in C:\Users\elkaa\Desktop\Source\IL2CPU\source\IL2CPU.Debug.Symbols\SQL.cs:line 49
1>C:\Users\elkaa\.nuget\packages\cosmos.build\0.1.0-localbuild20210416035217\build\Cosmos.Build.targets(258,9): error MSB4018:    at IL2CPU.Debug.Symbols.DebugInfo.CreateIndexes() in C:\Users\elkaa\Desktop\Source\IL2CPU\source\IL2CPU.Debug.Symbols\DebugInfo.cs:line 113
1>C:\Users\elkaa\.nuget\packages\cosmos.build\0.1.0-localbuild20210416035217\build\Cosmos.Build.targets(258,9): error MSB4018:    at IL2CPU.Debug.Symbols.ObjDump.ExtractMapSymbolsForElfFile(String debugFile, String mapFile) in C:\Users\elkaa\Desktop\Source\IL2CPU\source\IL2CPU.Debug.Symbols\ObjDump.cs:line 131
1>C:\Users\elkaa\.nuget\packages\cosmos.build\0.1.0-localbuild20210416035217\build\Cosmos.Build.targets(258,9): error MSB4018:    at Cosmos.Build.Tasks.ExtractMapFromElfFile.Execute() in C:\Users\elkaa\Desktop\Source\Cosmos\source\Cosmos.Build.Tasks\ExtractMapFromElfFile.cs:line 79
1>C:\Users\elkaa\.nuget\packages\cosmos.build\0.1.0-localbuild20210416035217\build\Cosmos.Build.targets(258,9): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
1>C:\Users\elkaa\.nuget\packages\cosmos.build\0.1.0-localbuild20210416035217\build\Cosmos.Build.targets(258,9): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__26.MoveNext()
1>Done building project "CosmosKernel1.csproj" -- FAILED.
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========

As you can see, it displays this error when we try to extract a map file. Elf doesn't support map files in a modern build of Nasm and Yasm. So, should we remove the ExtractFromMapElf? Also, what is the need for extracting map files in Cosmos? If we should remove it, where is it located and how would we remove it properly? This is the only thing that is stopping us from successfully compiling an OS in an elf format. Should we remove the ability to extract map files? Or is it necessary, in order for us to successfully create an ISO image?

@elkaamee326
Copy link
Contributor Author

Okay, I fixed the error. Finally, Yasm supports elf!

@charlesbetros
Copy link
Member

There is still a map file for with address/method names?

Does bin also still work?

Debugging is working in visual studio? (ie. Breakpoints)

@elkaamee326
Copy link
Contributor Author

elkaamee326 commented May 22, 2021

For some reason, .bin doesn't work and it is corrupt. It only compiles with elf. This was caused from some problems with IL2CPU. I will close this pull request because it was very difficult to get this to work and it was very tedious. Also, trying to make this work was extremely difficult.

Also, I don't know why but with a .bin compilation with IL2CPU, it

Everytime I would try to do a .bin compilation with Cosmos, it would successfully build but this would also happen when I try to debug / run an OS with Cosmos:

image

This is my build log when I try to compile with .bin:
BuildLog.txt

As you can see, there are many of warning: value does not fit in 16 bit field and warning: value does not fit in signed 16 bit field when I try to compile it. I know IL2CPU is the culprit because I even tried to compile an .asm file that was generated from an older version of IL2CPU without the error but when I attempted to compile it with the new version of IL2CPU, all those random warnings appeared. Also, the only modifications I made to the program were that I had to remove all of the ELF_COMPILATION #ifndef and #ifdef statements so that elf can work with Yasm. I don't know what the exact problem is, but there is a bug somewhere in IL2CPU I believe.

@quajak quajak reopened this May 22, 2021
@quajak
Copy link
Member

quajak commented May 22, 2021

Thank you for all your work on this issue. Due to how much there is to gain from switching to yasm, I would like to keep this pr open, even if you are no longer interested in working on it, which is understandable. If I have time, I will take a look and see if I can figure out what is needed to get .bin running.

@charlesbetros
Copy link
Member

I tried nasm and yasm side by side like this:

nasm.exe -g null -f bin -DBIN_COMPILATION -O0 Kernel_bin.asm
nasm.exe -g dwarf2 -f elf -DELF_COMPILATION -O0 Kernel_elf.asm

yasm.exe -g null -f bin -DBIN_COMPILATION -O0 Kernel_bin.asm
yasm.exe -g dwarf2 -f elf -DELF_COMPILATION -O0 Kernel_elf.asm

I got a single warning about globals not supported using bin format, but I didn't have to change anything in the asm. The output and map file were created with no issues.

@elkaamee326
Copy link
Contributor Author

elkaamee326 commented May 25, 2021

Did you make any modifications to your installed version of IL2CPU itself to make it work? Because, when I tried to make it work, Cosmos just corrupted it somehow by displaying a bunch of warnings and I don't know why. Like those .bin warnings... Interesting...

@charlesbetros
Copy link
Member

charlesbetros commented May 25, 2021 via email

@MishaTy
Copy link
Contributor

MishaTy commented Nov 7, 2021

Not needed anymore because of #1993

@MishaTy MishaTy closed this Nov 7, 2021
@valentinbreiz valentinbreiz reopened this Feb 8, 2022
@valentinbreiz
Copy link
Member

Now works using multiboot2, no more invalid multiboot image

MishaTy
MishaTy previously approved these changes Feb 8, 2022
@valentinbreiz valentinbreiz merged commit c0b4cc7 into CosmosOS:master Feb 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants