Advanced Paging

[utterances-bot]

This is a general purpose comment thread for the “Advanced Paging” post.

[lilyball]

BTW there seems to be a discussion on Hacker News about how recursive page tables are unnecessary. I'm not qualified to opine here, and I haven't actually even gone through this post yet, I just wanted to make sure you saw it.

[wwestlake]

Fantastic article! This is the best explanation of page tables I have read, great examples with a few typos I was able to correct easily so code works.

Thanks :)

[phil-opp]

@wwestlake Thanks so much!

@lilyball Thanks for the pointer! I'll post an answer over there. Edit: See #545

[jeehoonkang]

Thanks for the great post!

I wonder if how the kernel stack is initialized in the page table. As far as I understand, the kernel stack should already be initialized before _start() is called, because the stack is used by the function. Then it should be the boot loader that initializes the kernel stack in the page table, right?

[phil-opp]

@jeehoonkang Yes, that's completely correct. The stack initialization happens here.

[lilyball]

In the Translating Addresses section, the definition of translate_addr was a little confusing. It only made sense when I realized that the level_4_table_addr had to be 0o_177777_777_777_777_777_0000.

More generally, that function makes the assumption that the level 4, 3, and 2 index portions of level_4_table_addr must all be 0o777, otherwise the sign extension is broken, which means level_4_table_addr itself must be >= 0o_177777_777_777_777_400_0000 (alternatively it must be <= 0o_000000_000_000_000_377_7777). More generally, even if the sign extension is fixed, it requires that the level 2, 3, and 4 indices all be the same recursive index.

The fact that it relies on these unstated assumptions is what made it confusing for me.

[lilyball]

In Creating a new Mapping, rustc tells me the unsafe block in the following is unnecessary:

    let map_to_result = unsafe {
        recursive_page_table.map_to(page, frame, flags, frame_allocator)
    };

And indeed the documentation for map_to that you link to does not say it's an unsafe fn.

[lilyball]

Looking further, it appears that version 0.4.0 of the x86_64 crate made this function unsafe, but we're currently using version 0.3.5.

[lilyball]

Ah, now that I've read further, you tell me to update to 0.4.0 for an unrelated issue.

[phil-opp]

@lilyball Thanks for the feedback!

Ah, now that I've read further, you tell me to update to 0.4.0 for an unrelated issue.

Yeah, I updated the blog to version 0.4.0 a few days before publishing the post. I added a note to the post in 8306f3e. Sorry for the confusion.

In the Translating Addresses section, the definition of translate_addr was a little confusing. It only made sense when I realized that the level_4_table_addr had to be 0o_177777_777_777_777_777_0000. […]

Good points! I rewrote that section in 6fe3313, I hope that it is more understandable now.

[mental32]

Absolutely brilliant! when do you think the second edition will be complete? @phill-opp

[phil-opp]

@mental32 Thank you! Depends what you mean with "complete". There is only a single topic from the first edition missing in the second edition (kernel heap). But of course I will continue the series from there and explore new topics such as multithreading, multiprocessing, and userspace programs. There's no shortage on interesting topics, so I think I'll just keep going as long as I have the necessary time :).

[mental32]

Ah yes I do suppose "correct" is relative, I suppose I mean topics like filesystems, POSIX compliance, porting libc, multitasking.

Also it would be cool to see how a bootloader is written, while I have no issues with bootimage for now it'd be interesting to see it be done manually. Do you have a roadmap or something setup where I can see what topics you plan on covering?

[mental32]

Also would version of this edition that can be downloaded locally, and distributed easily like a pdf, be avaliable?

[phil-opp]

@mental32 I don't have a complete roadmap. I have concrete plans for the next two or three posts (kernel heap, multitasking, multi-core), but after that I don't know yet. A post on the bootloader is planned too, I just need to find some time to write it.

Also would version of this edition that can be downloaded locally, and distributed easily like a pdf, be avaliable?

We don't have a PDF version at the moment. Maybe I'll create one when there is more content.

[Rua]

Will the concept of a "higher half" kernel come into play at some point in your tutorials? I'm very interested in how that would be done for a Rust kernel!

[phil-opp]

@Rua Not until we add support for userspace processes because it only becomes relevant then. But I'm not sure whether address space sharing is still the way to go given the Spectre vulnerability.

Putting the kernel in the higher half should be relatively straightforward with our bootloader because the bootloader already maps the kernel to its specified virtual addresses. So you would just need to create a linker script that sets the starting virtual address to your desired address and specify the linker script in your target configuration ("pre-link-args": ["-Tlink.ld"]).

The linker script could look like this:

SECTIONS {
    . = 0xffffff0000100000; /* the desired start address */
}

Let me know if you need further help!

[tanguc]

Really nice explanation, it's awesome, thank you :)

[phil-opp]

@tanguc Thanks!

[tychota]

Hi !

(I came here because your blog is referenced by utteranc.es as a good usage of their comment system and I was needing one for mine. Now I'm writing an OS in rust ! ^^)

create_mapping was never defined, but create_example_mapping was. Maybe it is a typo ?

When building with create_example_mapping, without further modification, I got a page fault. 😱

When I modify the example mapping as:

pub fn create_example_mapping(
  recursive_page_table: &mut RecursivePageTable,
  frame_allocator: &mut impl FrameAllocator<Size4KiB>,
) {
  use x86_64::structures::paging::PageTableFlags as Flags;

- let page: Page = Page::containing_address(VirtAddr::new(0x1000));
+ let page: Page = Page::containing_address(VirtAddr::new(0xdeadbeaf900));
  let frame = PhysFrame::containing_address(PhysAddr::new(0xb8000));
  let flags = Flags::PRESENT | Flags::WRITABLE;

  let map_to_result = unsafe { recursive_page_table.map_to(page, frame, flags, frame_allocator) };
  map_to_result.expect("map_to failed").flush();
}

it works OK. 😍

Would you mind if I create a PR to fix that typos, I can do that tomorrow or so ?

Anyway, thanks for the blog. Long time I haven't done low-level stuff and it is really a pleasure to do it with Rust (I destroyed a board doing this https://github.com/tychota/ADXL345/blob/master/i2c-adxl345-input.c ^^). Your explanations are really valuable too.

[phil-opp]

Hi @tychota,

(I came here because your blog is referenced by utteranc.es as a good usage of their comment system and I was needing one for mine. Now I'm writing an OS in rust ! ^^)

That's a good story :D. Utterances works great for me so far!

create_mapping was never defined, but create_example_mapping was. Maybe it is a typo ?

Yes, I think that's a typo. I think I renamed that function but apparently forgot to update some references. A PR that fixes this would be great!

When building with create_example_mapping, without further modification, I got a page fault. 😱

When I modify the example mapping as: […] it works OK. 😍

We first map page 0x1000 and later 0xdeadbeaf000. The change from 0x1000 to 0xdeadbeaf000 is mentioned in the post right before the Boot Information section.

Anyway, thanks for the blog. Long time I haven't done low-level stuff and it is really a pleasure to do it with Rust (I destroyed a board doing this https://github.com/tychota/ADXL345/blob/master/i2c-adxl345-input.c ^^). Your explanations are really valuable too.

You're welcome, I'm glad that you like it!

[AntoineSebert]

I know it sounds silly, but I get clippy warnings because 0xdeadbeaf000 and 0xf021f077f065f04e should be respectively written 0x0dea_dbea_f000 and 0xf021_f077_f065_f04e.
Do you think it needs a PR ?

[phil-opp]

I don't think that it's an improvement for the deadbeaf address. But it definitely makes 0xf021f077f065f04e more readable.

[AntoineSebert]

Brace yourself, PR is coming.