New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: runtime: allow access to runtime.moduledata.text #28864
Comments
More detail would be helpful. You can access the address of a string easily today using
|
(CC @aclements @randall77 for |
I see that my original description was not very helpful. I have modified it slightly. After I get the address of the string I want to use the address as an index in my cache/hashtable.
If I know the address of the text section I know what is the base address of all constant data in the executable and (theoretical maximum) size of the constant data. With the base address and size known I know how large my cache should be (how many strings should fit). My hashtable key is a trivial (addressOfTheString-TextBaseAddress)/8 where 8 is an alignment for strings in Go (x64?). A key in my cache is a string offset in the text section. The cache in this case is a simple array. No hashing is involved.
A binary log like my small proof of concept here https://github.com/larytet/binlog/blob/master/binlog.go#L113 could use address of the text section moduledata.text instead of relying on /procfs/self/maps
Functions doing operations with strings which change rarely can benefit. The functions can easily cache results and retrieve the results quickly. A good example is log.Printf(). API of log could "compile" the format strings and, arguably, process the arguments faster.
This is not an easy question. Often the linkfile adds a global variable pointing to the ".text". The linkfile/linker venue is one approach. I can hash the 64 bits virtual address of the string. This is slower than calculating the offset, but probably fast enough. Distributions of keys can be a problem. String offset from the .text base promises the perfect distribution and a sparse hashtable - better performance, worse memory footprint. I can hash the string itself. This is what I do when I see the address of the format string does not fit the address range. This is the slowest option. Lookups in a small map alone will take 40-50ns/op. runtime.moduledata is an absolutely gold mine. Bootstrap (?) does a lot of work to collect all the priceless data in one place. I tried to |
Your suggestion relies on specific details about the current runtime implementation that are not guaranteed to be true for all implementations. If we exposed this information, and people relied on it in the way you suggest, we would be tying the hands of all later implementations. I think this would be a bad idea. If you want to be portable to other platforms and other versions of Go, use a map with a string key. If you don't care about being portable, use go:linkname and type reflection. |
I need an instance of the type to use the type reflection, don't I? |
Yes, I suppose you're right. Sorry. |
For the reasons Ian pointed out, we're definitely not going to expose a public API around You're right that you can't linkname If you're willing to linkname things, however, you can get the base of the text segment without //go:linkname text runtime.text
var (
text struct{}
textAddr = uintptr(unsafe.Pointer(&text))
) The |
@aclements , This is a great tip. Do I have a way to get the size of the .text section as well? |
Do you have to use the start of .text as the base? Would any fixed address in the text section, like the address of |
@cherrymui, I am using the offset as an index in the array of processed (compiled) format strings. An address of the first constant string in the executable would be the best and ensure that my index starts from zero. The sad truth about the production logging is that
Most choose not to log, instead enable the logs manually when required. |
I guess I will keep using /proc/self/maps |
I agree that it is probably ideal to use the address of first string, but it should be suffice to use any given text symbol. Another possibility of using just public API: using |
@cherrymui, I still need the size of the .text section re ELF this is what https://github.com/martende/restartable does Platform dependent, lot of code. |
Looks like I end up with duplicating moduledata in my code anyway. I want to find all Go modules the executable (ELF) depends on, go/parse the relevant Go source files, go/ast all calls to binlog.Log(), collect the arguments, hash the format strings. The first step - getting list of modules from the executable - has more than one possible approach
|
I keep format strings (things similar to the first argument of fmt.Sprintf) in a hash table. I use address of the string instead of hashing the string. I assume that the strings are const data and stored in the text segment of the executable. In the existing code I rely on /procfs/self/maps which works only in Linux. Given the starting address of the text segment I can calculate the string offset and use the value as an index in the cache table.
An API returning runtime.moduledata.text can help to improve performance in functions handling strings. Example of such API is custom versions of fmt.Sprintf()
The text was updated successfully, but these errors were encountered: