Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lib/uklibid: Library identifiers and SUBBUILD build system feature #938

Closed
wants to merge 15 commits into from
Closed

lib/uklibid: Library identifiers and SUBBUILD build system feature #938

wants to merge 15 commits into from

Conversation

skuenzer
Copy link
Member

@skuenzer skuenzer commented Jun 8, 2023 •
edited by nderjung
Loading

Base target

  • Architecture(s): [all]
  • Platform(s): [all]
  • Application(s): [all]

Additional configuration

  • CONFIG_LIBUKLIBID=y

Description of changes

This PR introduces lib/uklibid which is a library that generates at compile-time unique IDs for each library. These IDs are starting from 0 and linearly increase so that they can be used for table lookups with O(1) complexity. lib/ukdebug and lib/ukstore are first users of this library and adopted accordingly.

As required ground work, this PR also introduces a SUBBUILD feature to the build system. This feature adds the ability to place generated files (e.g., pre-processed, compiled) in a subdirectory of the corresponding library build directory instead of just placing them in the top-level directory. For this purpose, the build system looks for a file-scoped variable with the suffix _SUBBUILD. Such a variable can contain a path that is created below the library build directory.

Example (Makefile.uk):

# Create `myheader.h` under `build/libmylib/include/uk/`
LIBMYLIB_SRCS-y += $(LIBMYLIB_BASE)/myheader.m4>.h
LIBMYLIB_MYHEADER_SUBBUILD += include/uk

@skuenzer skuenzer requested review from a team as code owners June 8, 2023 01:44
@skuenzer skuenzer requested a review from michpappas June 8, 2023 01:45
@unikraft-bot unikraft-bot added area/lib Internal Unikraft Microlibrary lang/c Issues or PRs to do with C/C++ lib/syscall_shim lib/ukdebug labels Jun 8, 2023
@michpappas michpappas mentioned this pull request Jun 8, 2023
Closed
4 tasks
@razvand razvand removed request for a team June 12, 2023 11:15
@razvand razvand added this to the v0.14.0 (Prometheus) milestone Jun 12, 2023
This was referenced Jun 13, 2023
Open
Closed
michpappas
michpappas requested changes Jun 13, 2023
View reviewed changes
Copy link
Member

@michpappas michpappas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @skuenzer nice PR! 🚀 Just a couple of minor issues and some clarifications from my side.

support/build/Makefile.rules Show resolved Hide resolved
support/build/Makefile.rules Show resolved Hide resolved
support/build/Makefile.rules Show resolved Hide resolved
lib/uklibid/include/uk/libid.h Outdated Show resolved Hide resolved
lib/uklibid/include/uk/libid.h Outdated Show resolved Hide resolved
lib/uklibid/include/uk/libid.h Show resolved Hide resolved
lib/uklibid/include/uk/libid.h Show resolved Hide resolved
lib/uklibid/libname.c Outdated Show resolved Hide resolved
lib/uklibid/include/uk/libid.h Outdated Show resolved Hide resolved
lib/ukstore/Makefile.uk Show resolved Hide resolved
michpappas
michpappas approved these changes Jun 30, 2023
View reviewed changes
Copy link
Member

@michpappas michpappas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks! @razvand it's on you now.

Reviewed-by: Michalis Pappas michalis@unikraft.io

@michpappas michpappas requested a review from kubanrob July 20, 2023 17:00
kubanrob
kubanrob reviewed Jul 21, 2023
View reviewed changes
lib/uklibid/Makefile.uk Outdated Show resolved Hide resolved
@skuenzer skuenzer force-pushed the skuenzer/uklibid branch 3 times, most recently from 6e889c7 to b9bce31 Compare August 4, 2023 09:06
@razvand razvand requested a review from kubanrob August 5, 2023 06:33
@kubanrob
Copy link
Contributor

kubanrob commented Aug 6, 2023

I see the same error as integration / libc-test on arm64: E tar: /github/workspace/_helloworld/.unikraft/build/libmusl/origin: Cannot open: No such file or directory.

@skuenzer
lib/uklibid: Generate libraries.in
04ab0c7 
`libraries.in` contains a list of currently selected libraries.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
@skuenzer
lib/uklibid: Compile-time mapping of library names to IDs
f47eef4 
This commit introduces macros for mapping a library name to a compile-time
unique identifier with the pre-processor. The library IDs are starting from
`0` and increase linearly. The purpose is that such an ID can be directly
used as index for looking up a corresponding entry within an array.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
@skuenzer
lib/uklibid: Run-time name resolution of library IDs
5761ccb 
This commit introduces the function `uk_libname()` that resolves a library
ID to its library name at run-time. The function returns a reference to a
corresponding C-string with a lookup table.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
@skuenzer
lib/uklibid: Run-time mapping of library name to identifier
35d757a 
This commit introduces the function `uk_libid()` that finds the library ID
for a given library name at run-time.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
@skuenzer
lib/ukdebug: Use lib/uklibid for library names
af381ba 
Every invocation of any of the provided print function macros by
`lib/ukdebug` (`uk_printk()`, `uk_printd()`), caused placing a C-string
containing the library name in the sources. This commit replaces this
C-string with a library identifier which is just an unsigned integer. The
ID is only resolved into a name in the actual print format function
(`_vprint()`). The resolution is done by `uk_libname()` that is using the
ID for a table lookup (O(1)).

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
@skuenzer
lib/ukstore: Re-use libraries.in from lib/uklibid
73bdf2a 
Instead of generating own library identifiers, this commit adopts
`lib/ukstore` to use the same library catalog from `lib/uklibid`. At the
same time, this commit adopt `Makefile.uk` to make use of the `SUBBUILD`
feature of the build system. This is done for simplifying Makefile.uk and
to let the build system natively generate each dynamic headers of
`lib/ukstore`.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
@skuenzer
lib/ukstore: Use lib/uklibid for library ID mapping
4ad9220 
Instead of implementing own library ID mapping macros and functions, this
commit adopts `lib/ukstore` to use `lib/uklibid` for this purpose.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
@skuenzer
lib/uklibid: Supporting pre-built libraries
948604f 
This commit introduces a clearer separation between runtime and
compile-time resolution of library identifiers with the API. This is done
so that libraries can potentially be built off-tree and shipped in binary
form. Each library's own identifier (`uk_libid_self()`) is stored in a
constant variable that is resolved during link-time. The owner of this
variable is `uklibid`, so the identifier is external to a binary library.

A binary delivery of a library can be created with Unikraft (for example:
`make libbinary`) and the resulting library object (likely under
`build/libbinary.o`) can be delivered with a `Config.uk` like:
```KConfig
config LIBBINARY
	bool "Binary example library"
	default n
```

and a `Makefile.uk` like:
```Makefile
UK_OLIBS-$(CONFIG_LIBBINARY) += $(IMPORT_BASE)/libbinary.o
UK_LIBID_EXTRA-$(CONFIG_LIBBINARY) += libbinary
```

Please note that the library must be registered directly in `UK_OLIBS-y`;
`addlib` or `addlib_s` cannot be used in such cases.
Please also note that such a library should restrict dependencies on other
libraries and their configuration options. In particular, the use of macros
and inline functions must be handled with care.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
@skuenzer
Copy link
Member Author

skuenzer commented Aug 8, 2023

As we discussed offline, this release now includes support for shipping precompiled libraries, although uklibid identifiers are used. For a more detailed explanation, see commit 948604f.

kubanrob
kubanrob approved these changes Aug 9, 2023
View reviewed changes
Copy link
Contributor

@kubanrob kubanrob left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thank you @skuenzer!

Reviewed-by: Robert Kuban robert.kuban@opensynergy.com

razvand
razvand approved these changes Aug 9, 2023
View reviewed changes
Copy link
Contributor

@razvand razvand left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved-by: Razvan Deaconescu razvand@unikraft.io

unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
build: Only first-level subdirectories with mk_sub_build_dir
b8872a6 
We have two pairs of functions that are internally used to create the
directory structure the build output directory (normally `build/`):
- The function `mk_sub_build_dir` is intended to create only the first
  level subdirectory under the build output directory for a library.
  `sub_build_dir` is a companion function that returns the full path
  of such a first-level subdirectory for a given library name.
- The function `mk_sub_libbuild_dir` is intended to create further
  subdirectories under a library build directory that was created by
  `mk_sub_build_dir`. This function allows multiple levels of
  subdirectories (example: to create an header include folder:
  'include/uk/bits').
  `sub_libbuild_dir` is the companion function that returns the
  full path of such a library subdirectory.

This commit corrects the implementation so that this behavior is
enforced.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
build: Introduce *_SUBBUILD for compilation units
829652a 
This commit adds to the build system the ability to place generated
files (e.g., pre-processed, compiled) in a subdirectory of the
corresponding library build directory (instead of just placing them in the
top-level directory). For this purpose, the build system looks for a
file-scoped variable with the suffix `_SUBBUILD`. Such a variable can
contain a path that is created below the library build directory.

Example (`Makefile.uk`):
 # Create `myheader.h` under `build/libmylib/include/uk/`
 LIBMYLIB_SRCS-y += $(LIBMYLIB_BASE)/myheader.m4>.h
 LIBMYLIB_MYHEADER_SUBBUILD += include/uk

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
build: Support generating .uk files with preprocessor
48e3311 
This commit registers the `.uk` file extension as known extension that can
be generated by the build system with a pre-processor (e.g., M4, AWK).

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
build: Provide $(IMPORT_BASE) to Makefile.uk files
a883437 
This commit makes the variable `$(IMPORT_BASE)` usable in `Makefile.uk`
files. This is needed to include libraries that do not use the `addlib`
or `addlib_s` registration functions. An example are libraries that are
shipped in binary form.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
lib/syscall_shim: Utilize SUBBUILD feature within Makefile.uk
23fe2fb 
This commit utilizes the recently added `SUBBUILD` feature of the build
system to natively generate all dynamic headers of the library. This has
several reasons:
 1) The `Makefile.uk` of `syscall_shim` becomes more readable
 2) No custom rule for creating the subdirectory structure is needed
 3) No custom `awk`-rules are needed for generating the headers
 4) The build system can take care of handling build order dependencies

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
lib/uklibid: Add library skeleton
8999cd4 
libuklibid will provide generated identifiers for each selected library.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
lib/uklibid: Generate libraries.in
57f5e08 
`libraries.in` contains a list of currently selected libraries.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
lib/uklibid: Compile-time mapping of library names to IDs
3fd3b22 
This commit introduces macros for mapping a library name to a compile-time
unique identifier with the pre-processor. The library IDs are starting from
`0` and increase linearly. The purpose is that such an ID can be directly
used as index for looking up a corresponding entry within an array.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
lib/uklibid: Run-time name resolution of library IDs
c122211 
This commit introduces the function `uk_libname()` that resolves a library
ID to its library name at run-time. The function returns a reference to a
corresponding C-string with a lookup table.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
lib/uklibid: Run-time mapping of library name to identifier
de6b018 
This commit introduces the function `uk_libid()` that finds the library ID
for a given library name at run-time.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
lib/ukdebug: Use lib/uklibid for library names
5971b59 
Every invocation of any of the provided print function macros by
`lib/ukdebug` (`uk_printk()`, `uk_printd()`), caused placing a C-string
containing the library name in the sources. This commit replaces this
C-string with a library identifier which is just an unsigned integer. The
ID is only resolved into a name in the actual print format function
(`_vprint()`). The resolution is done by `uk_libname()` that is using the
ID for a table lookup (O(1)).

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
lib/ukstore: Re-use libraries.in from lib/uklibid
f866f3f 
Instead of generating own library identifiers, this commit adopts
`lib/ukstore` to use the same library catalog from `lib/uklibid`. At the
same time, this commit adopt `Makefile.uk` to make use of the `SUBBUILD`
feature of the build system. This is done for simplifying Makefile.uk and
to let the build system natively generate each dynamic headers of
`lib/ukstore`.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
lib/ukstore: Use lib/uklibid for library ID mapping
2a39823 
Instead of implementing own library ID mapping macros and functions, this
commit adopts `lib/ukstore` to use `lib/uklibid` for this purpose.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
unikraft-bot pushed a commit that referenced this pull request Aug 9, 2023
@skuenzer @unikraft-bot
lib/uklibid: Supporting pre-built libraries
7815d0b 
This commit introduces a clearer separation between runtime and
compile-time resolution of library identifiers with the API. This is done
so that libraries can potentially be built off-tree and shipped in binary
form. Each library's own identifier (`uk_libid_self()`) is stored in a
constant variable that is resolved during link-time. The owner of this
variable is `uklibid`, so the identifier is external to a binary library.

A binary delivery of a library can be created with Unikraft (for example:
`make libbinary`) and the resulting library object (likely under
`build/libbinary.o`) can be delivered with a `Config.uk` like:
```KConfig
config LIBBINARY
	bool "Binary example library"
	default n
```

and a `Makefile.uk` like:
```Makefile
UK_OLIBS-$(CONFIG_LIBBINARY) += $(IMPORT_BASE)/libbinary.o
UK_LIBID_EXTRA-$(CONFIG_LIBBINARY) += libbinary
```

Please note that the library must be registered directly in `UK_OLIBS-y`;
`addlib` or `addlib_s` cannot be used in such cases.
Please also note that such a library should restrict dependencies on other
libraries and their configuration options. In particular, the use of macros
and inline functions must be handled with care.

Signed-off-by: Simon Kuenzer <simon@unikraft.io>
Reviewed-by: Michalis Pappas <michalis@unikraft.io>
Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com>
Approved-by: Razvan Deaconescu <razvand@unikraft.io>
Tested-by: Unikraft CI <monkey@unikraft.io>
GitHub-Closes: #938
@unikraft-bot unikraft-bot added the ci/merged Merged by CI label Aug 9, 2023
@nderjung nderjung added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Aug 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@razvand razvand razvand approved these changes

@michpappas michpappas michpappas approved these changes

@kubanrob kubanrob kubanrob approved these changes

Assignees

@razvand razvand

Labels
area/lib Internal Unikraft Microlibrary ci/merged Merged by CI lang/c Issues or PRs to do with C/C++ lib/syscall_shim lib/ukdebug release-note Denotes a PR that will be considered when it comes time to generate release notes.
Projects
Unikraft Core Roadmap
Status: ✅ Done!
Milestone
v0.14.0 (Prometheus)
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@skuenzer @kubanrob @nderjung @unikraft-bot @razvand @michpappas