Non functional change.

Tuples become hard to read after you get enough entries.

We had two codepaths with almost duplicate code. This commit factors out
the common code.

Non functional change.

Just some small code cleanup.

This commit adds support for duplicated USDT markers. Duplication in
this context means identical provider and name. See the later runtime
test for an example of how this can occur.

With this commit, bpftrace will generate separate programs for each
duplicated marker. We need to generate separate programs because the
location of the usdt arguments may not be the same in each call site.
The later runtime test does not reproduce this behavior but I've
observed this happening in production before.

This commit sets up all the scaffolding for actual implementation.

This helper is used to determine if a memcpy is necessary or if a simple
load will work.

Before:
```
stdin:1:9-16: ERROR: Can not access field '' on expression of type 'int64'
BEGIN { (1+2).2 }
        ~~~~~~~
```

After:
```
stdin:1:9-16: ERROR: Can not access index 2 on expression of type 'int64'
BEGIN { (1+2).2 }
        ~~~~~~~
```

This commit implements codegen for tuple creation and tuple indexing.

Note that tuples are allocated on the stack because we can't really
contain aggregate types in registers.

Have `map_value_to_str` take a SizedType instead of an `IMap` reference
so we can later reuse this method to print tuple elements.

Support for both text and json output

Forgot to add checks for <sys/sdt.h> headers.

Arrays used the `size`, `elem_type` and `pointee_size` fields to store
array type information. This is confusing as `pointee_size` indicates a
pointer and also incomplete as an array of pointers cannot be stored
without losing pointee size information.

This fixes that by storing the complete type (`SizedType *`) instead of
just a few fields.

Use "bpftrace" instead of '"bpftrace"'. Previously this causes build
error if included header files use KBUILD_MODNAME.

Split a long function and call functions only when necessary

BTF::c_def() now accepts enum name, such as "enum bpf_prog_type"

'-l' option now can take struct/union/enum name and dump its definition
using BTF. For example:

```
% sudo ./src/bpftrace -l "enum bpf*"
enum bpf_access_type
enum bpf_adj_room_mode
enum bpf_arg_type
enum bpf_attach_type
enum bpf_cgroup_storage_type
[...]
```

and

```
% sudo ./src/bpftrace -lv "struct path"
BTF: using data from /sys/kernel/btf/vmlinux
struct path {
        struct vfsmount *mnt;
        struct dentry *dentry;
};
```

Now `bpftrace -l -p 11234` only lists uprobe and usdt probes.
`bpftrace -l "k:*" -p 11234` lists kprobe, which is the same as before.

We haven't seen any OOM issues in a while so I suspect either our type
fixes in bpftrace or upstream llvm changes have resolved the issue.

This closes #1355.

GetElementTy() returns a pointer so we need to deref. We don't expect
GetElementTy() to return null b/c an array type should have the element
type set.

Helps with debugging things on the issue tracker.

Not sure how it used to compile. Magic, I guess.

Commit f05b4cd ("Introduce Type::ctx") introduced a `ctx` type
for values that come from the bpf prog context. This subtly broke ntop()
because ntop was not yet aware of Type::ctx.

Fix this by relaxing semantic analyser check to include Type::ctx.
Similarly relax for codegen.

I think to prevent job `irc` from being skipped you need to specify the
dependency.

See
https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idneeds

I think TLS support from notify-irc is kinda broken.

This commit just cut and pastes code around. It's nicer to have
small, orthogonal modules.

Keeping unused code around is asking for bit rot and bugs. Also hide
some methods that are used for internal caching.

USDTHelper caching logic was previously incorrect. It would mark the
cache as loaded if any pid or path was processed. This would cause
scripts that place usdt probes on different binaries to silently fail
(because a default usdt_probe_entry would be returned).

This commit breaks up the cache by pid or by path. Also switch
usdt_provider_cache to an unordered_map b/c we don't need ordering and
unordered data structures are faster.

Next commit will get rid of the silent failure behavior.

It's good to have an error reporting mechanism. Returning a default
value caused a lot of headache when trying to debug usdt lookup failures
that were fixed in the previous commit.

Before:

    $ sudo ./build/src/bpftrace -e 'usdt:/usr/bin/python3:python:asdf { 1 }'
    terminate called after throwing an instance of 'std::runtime_error'
      what():  Failed to find usdt probe: usdt:/usr/bin/python3:python:asdf
    [1]    865191 abort      sudo ./build/src/bpftrace -e

After:

    $ sudo ./build/src/bpftrace -e 'usdt:/usr/bin/python3:python:asdf { 1 }'
    Failed to compile: Failed to find usdt probe:
    usdt:/usr/bin/python3:python:asdf

This patch aims to replace the external tools which are used in runtime tests to trigger syscalls. It can generate sys_nanosleep, sys_open, sys_openat, sys_read and sys_execve.

There are two reasons why it is important not to rely on external tools to trigger syscalls. The first is that some tools are not always available on some systems. And the other is that, the tools don't essentially work to generate certain syscalls. It would be more portable if we can have an program working to provide these syscalls.

Some architecture like ppc64, don't use sync_file_range but
sync_file_range2. We can use a wildcard to get them both cover.

Fixes the following error:

$ /usr/share/bpftrace/tools/syncsnoop.bt
Attaching 7 probes...
open(/sys/kernel/debug/tracing/events/syscalls/sys_enter_sync_file_range/id): No such file or directory
Error attaching probe: tracepoint:syscalls:sys_enter_sync_file_range

The bpftrace man page uses the escape \' to presumably print quote, but this troff escape is meant to print an acute accent. Beside making the man page look strange, it also prevents copy-pasting examples.

This prevents curl from spamming the terminal.

This patch adds support for new time units us and hz for probe interval.

Such changes make the time units of interval consistent with those of profile.

This closes #1335 .

Before, I think we were intentionally leaking the bcc usdt context so
that usdt semaphore values would not go back to zero. Reason is that
bcc_usdt_close() decrements usdt semaphore value.

However, this causes issues if we don't eventually decrement the counter
as the tracee will forever be on more expensive code paths.

This commit defers semaphore decrement until probe is detached.

While debugging a high memory consumption issue, I noticed that a
USDT::Context object can take ~10M per instance [0]. Along with the new
--usdt-file-activation, bpftrace can potentially hold onto many dozens
of USDT:Context instances, causing memory issues.

This patch makes use of the new bcc usdt semaphore helpers (
iovisor/bcc#2953 ) to avoid holding onto
USDT::Context instances for the lifetime of the tracing session.

The new algorithm is:

1. create a USDT::Context
2. increment the semaphore count for the probe we care about
3. destroy the USDT::Context
4. repeat 1-3 for all probes we want to attach to
5. do our tracing
6. create a USDT::Context for the probe we care about
7. decrement the semaphore count
8. destroy the USDT::Context
9. repeat 6-8 for all the probes we're attached to

This approach also has the benefit of 1 USDT::Context instance being
alive at a time which can help keep memory high watermark low.

[0]: Through gdb single stepping and /proc/pid/status. Exact process is
not described here b/c memory usage probably varies based on tracee
binary.

These two functions add a lot of noise in the "before optimized" output,
which is annoying when debugging.

cf. [x86-64 struct pt_regs](https://github.com/torvalds/linux/blob/master/arch/x86/include/uapi/asm/ptrace.h).

Fixes the following errors.

```
% ./src/bpftrace -e 'k:f {  printf("%d\n", reg("fs_base")); }'
Attaching 1 probe...
Error loading program: kprobe:f (try -v)
```

```
% sudo ./src/bpftrace -e 'i:s:1 {  printf("%d, %d\n", reg("fs_base"), ctx->sample_period); }'
Attaching 1 probe...
1000000000, 1000000000
^C
```

Example:

```
% cmake ..
[...]
-- Found LLVM 10.0.1: /usr/lib/llvm-10/lib/cmake/llvm
```

This implements non-map prints as described in #356.

Note that non-map print()s are synchronous, meaning they do not
work the same as map print()s. I think this is a pretty nice feature.

Example:
```
# bpftrace -e 'BEGIN { $t = (1, "string"); print(123); print($t); print(comm) }'
Attaching 1 probe...
123
(1, string)
bpftrace
^C
```

We have a growing number of types in the type system so I may have
missed a corner case somewhere. I think I got most of it right, though.

This closes #356.

Fix a bad check in.

Issue:
- cpid is written to higher 32 bits in BE
- bpftrace -v -e 'BEGIN { printf("%d", cpid); }'
Bytecode:
0: (bf) r6 = r1
1: (b7) r1 = 0
2: (7b) *(u64 *)(r10 -8) = r1
last_idx 2 first_idx 0
regs=2 stack=0 before 1: (b7) r1 = 0
3: (b7) r2 = 122768
4: (63) *(u32 *)(r10 -8) = r2 <-----
5: (7b) *(u64 *)(r10 -16) = r1
(...)

Fix:
- Promote it to 64 bits.

Signed-off-by: Sumanth Korikkar <sumanthk@linux.ibm.com>

Issue:
- Length is not interpreted properly in buf builtin for BE.
- Format of buf: (length: 1 byte | buf: char *), Also see resolve_buf().

bpftrace -v -e 'kprobe:__sys_sendto { printf("hello: %r", buf((arg1),
4)); exit(); }'
Bytecode:
0: (bf) r6 = r1
1: (b7) r1 = 0
2: (7b) *(u64 *)(r10 -24) = r1
last_idx 2 first_idx 0
regs=2 stack=0 before 1: (b7) r1 = 0
3: (7b) *(u64 *)(r10 -16) = r1
4: (b7) r1 = 4
5: (7b) *(u64 *)(r10 -8) = r1  <--- Intepreted size as 8 bytes
6: (79) r3 = *(u64 *)(r6 +104)
7: (bf) r1 = r10
8: (07) r1 += -7
9: (b7) r2 = 4
10: (85) call bpf_probe_read#4
last_idx 10 first_idx 0
regs=4 stack=0 before 9: (b7) r2 = 4
11: (61) r1 = *(u32 *)(r10 -8) <---- Here size + some bytes of buf is
                                     interpreted as zero for BE
12: (63) *(u32 *)(r10 -16) = r1
13: (71) r1 = *(u8 *)(r10 -4)
14: (73) *(u8 *)(r10 -12) = r1
15: (18) r7 = 0xffff9eac33c9f800
17: (85) call bpf_get_smp_processor_id#8
18: (bf) r4 = r10
19: (07) r4 += -24
20: (bf) r1 = r6
21: (bf) r2 = r7
22: (bf) r3 = r0
23: (b7) r5 = 16
24: (85) call bpf_perf_event_output
(...)

Fix:
- Fix it by casting length of buf to 1 byte.

Signed-off-by: Sumanth Korikkar <sumanthk@linux.ibm.com>

Issue:
- ./bpftrace -e '#include <linux/socket.h>
  BEGIN { @=ntop(AF_INET, 0x0100007f);}'
  @: ntop() got unsupported AF type: 0

Fix:
- aftype size is 64 bits in codegen. Hence cast aftype to 64bit

Signed-off-by: Sumanth Korikkar <sumanthk@linux.ibm.com>

Issue:
- 32 bit unop deferences does not work for Big endian
- Example: bpftrace -v -e 'uprobe:testprogs/uprobe_test:function1 {
                printf("0x%lx -- 0x%lx\n", *uaddr("GLOBAL_A"),
                *uaddr("GLOBAL_C")); }' fails.

Fix:
- Promote the expr in codegen to 64 bit. Integer types are already
  promoted to 64bits. Hence casting performed in get_arg_values() and
  codegen matches.

Signed-off-by: Sumanth Korikkar <sumanthk@linux.ibm.com>

Issue:
- 32 bit dereferences fails for Big endian
- Example: bpftrace -v -e 'tracepoint:random:urandom_read { $i =
  args->got_bits; printf("bits: %d\n", $i); if ($i == 24) { exit() } }'
  - Example: bpftrace -v -e 'struct Foo { int a; char b[10]; }
    uprobe:testprogs/uprobe_test:function2 { $foo = (struct Foo *)arg0;
    $foo2 = (struct Foo *)arg1; printf("%d\n", $foo->a) }' fails

Fix:
- Promote Fieldaccess expr_ for these cases to 64 bit.

Signed-off-by: Sumanth Korikkar <sumanthk@linux.ibm.com>

Issue:
bpftrace -v -e 'struct MyStruct { int y[4]; }
uprobe:./testprogs/array_access:test_struct { $s = (struct MyStruct *)
arg0; $x = $s->y[0]; printf("Result: %d\n", $x); exit(); }' fails

Bytecode:
0: (bf) r6 = r1
1: (79) r3 = *(u64 *)(r6 +40)
2: (bf) r1 = r10
3: (07) r1 += -16
4: (b7) r2 = 4  <--- Read four bytes (field.type.size is 4)
5: (85) call bpf_probe_read#112
last_idx 5 first_idx 0
regs=4 stack=0 before 4: (b7) r2 = 4
6: (b7) r1 = 0
7: (7b) *(u64 *)(r10 -8) = r1
last_idx 7 first_idx 0
regs=2 stack=0 before 6: (b7) r1 = 0
8: (61) r2 = *(u32 *)(r10 -16) <--- Reads the topmost bytes in BE. which
                                    is zero.
9: (63) *(u32 *)(r10 -8) = r2
(...)

Fix:
- Promote the ArrayAccess to 64 bit.

Signed-off-by: Sumanth Korikkar <sumanthk@linux.ibm.com>

Issue:
The first sarg value should be 0xfed. Only five reg args for s390x.

Fix:
Fix this in intptrcast tests

Signed-off-by: Sumanth Korikkar <sumanthk@linux.ibm.com>

Currently, the ternary operator can only works to evaluate strings and
integers. This tries to include builtin calls:
```
# bpftrace -e 'BEGIN { pid > 10000 ? printf("%s %d\n", comm, pid) : exit(); }'
```

Although if-else statement can serve the purpose, using ternary can be
more concise especially when users run bpftrace in comand line. In
bpftrace, builtin calls have `none` types so it makes sense for the
ternary operator to support them. Resolves #1333.

llvm-as emits warnings when the IR has typing issues:

```
 error: '%2' defined with type 'i8*'
  call void @llvm.lifetime.start.p0i8(i64 -1, i64* nonnull %2)
```

We can use this to validate that the IR we generate is valid and avoid
regressions (like the OOM error).

As the codegen reference output is LLVM 7 based this is currently only
enabled for LLVM 7. Testing the version specific things is not
necessary as we don't have any version specific logic in the codegen.

The tests that do not pass are ignored. Eventually we shouldn't have to
ignore any of them. Adding new entries to this list should be avoided,
unless the test overlaps with something currently on the list.

builtin_func_1.ll
builtin_func_uprobe_2.ll
call_buf_implicit_size.ll
call_buf_size_literal.ll
call_buf_size_nonliteral.ll
call_print_non_map.ll
call_printf.ll
if_else_printf.ll
if_nested_printf.ll
if_printf.ll

Related to #1091

Does the same thing and less code to maintain.

This helper makes a lot of assumptions about the input and probably
shouldn't be a public API if we can help it.

Also rename the lower level helper b/c it doesn't look like function
overloading works correctly with class member and non-class member.

Before, `find_wildcard_matches` would choke on lines in
available_filter_functions like:

    ehci_disable_ASE [ehci_hcd]

This patch has k[ret]probes ignore the module annotation.

This closes #1036 .

This enables printf, cat and system to have more than 7 arguments. It can be
more user-friendly to allow any reasonable number of arguments for these builtin
calls which use formatted strings. Resolves #1336.

print_map will do the same type checkings again.

…ctly

followed by any alphabetic characters since it would take all the following
letters as part of the specifiers. As an example:

```
bpftrace -e 'BEGIN { @start = nsecs; } i:s:1 { printf("Elapsed: %ds\n", (nsecs - @start)/1000000000); }'
stdin:1:35-95: ERROR: printf: Unknown format string token: %ds
BEGIN { @start = nsecs; } i:s:1 { printf("Elapsed: %ds\n", (nsecs - @start)/1000000000); }
```
I think it's good to fix it since I could not think of any obvious downsides.
This works to address this issue by changing the way `printf` parse the
format strings. It would also affect `cat` and `system` as well. With this PR:

```
bpftrace -e 'BEGIN { @start = nsecs; } i:s:1 { printf("Elapsed: %ds\n", (nsecs - @start)/1000000000); }'
Attaching 2 probes...
Elapsed: 1s
Elapsed: 2s
Elapsed: 3s
Elapsed: 4s
Elapsed: 5s
^C

@start: 2976003477708
```

This PR has been merged right after the 0.11 release so it shouldn't be
in the 0.11 release.

The implicit dependency is confusing for users who somehow have a
bpftrace built without libbpf.

This closes #1422 .

Some runtime tests in builtin and probe expect the system to generate vfs_read
events for them, which in some cases might cause timeout error. This commit
explicitly generates the event for the tests with testprog/syscall.

This introduces a new builtin function `strftime`. This function makes it
possible to print the current time with `printf`:

``` bpftrace -e 'i:s:1 { printf("%s\n", strftime("%H:%M:%S", nsecs)); }' ```

Although `time` can print current time on its own, printing timestamp with
`printf` can produce more concise and more readable output. Resolves #1337 .

Currently we test codegen after optimization. this has 2 issues:

We don't actually test the code we generate, we test what LLVM does to
it. This makes harder to relate a change in the codegen to a change in
the LLVM output.

We have to maintain more version specific codegen files as we cannot
rewrite big changes the optimizer has done.

Fixes: #1368

When building with clang a few codegen tests fail due to argument
reordering. This normally doesn't show up as the optimizer step fixes
that, but as we're now testing before optimization it get's in our way.

This means that a "complex" call like:
```
b_.CreateUDiv(b_.CreateSub(b_.CreateLoad(value_alloc), b_.CreateLoad(min_alloc)), b_.CreateLoad(step_alloc))
```

Will have to be split into:
```
Value *step = b_.CreateLoad(step_alloc);
Value *min = b_.CreateLoad(min_alloc);
Value *val = b_.CreateLoad(value_alloc);
Value *div3 = b_.CreateUDiv(b_.CreateSub(val, min), step);
```

To ensure that all arguments are always evaluated in the same order.

The move to unoptimized codegen uncovered more type issues that we have
to fix. Fixing those now will blow up this PR even more and make it hard
to review so lets just ignore them for now.