Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to run cilium with clang-3.9 #172

Closed
aanm opened this issue Oct 20, 2016 · 2 comments
Closed

Unable to run cilium with clang-3.9 #172

aanm opened this issue Oct 20, 2016 · 2 comments
Labels
kind/bug This is a bug in the Cilium logic.

Comments

@aanm
Copy link
Member

aanm commented Oct 20, 2016

Steps to reproduce.

  1. Replace clang in root's Dockerfile with 3.9.0.
  2. Run the docker-compose example provided here: https://github.com/cilium/cilium/tree/master/examples/docker-compose
  3. The error happens on the first container started. For example: docker run -d --name wine --net cilium --label io.cilium.service.wine noironetworks/nettools sleep 30000
cilium_1         | Join EP id=29898 ifname=lxcd7915
cilium_1         | 
cilium_1         | Prog section 'from-container' rejected: Permission denied (13)!
cilium_1         |  - Type:         3
cilium_1         |  - Instructions: 3276 (0 over limit)
cilium_1         |  - License:      GPL
cilium_1         | 
cilium_1         | Verifier analysis:
cilium_1         | 
cilium_1         | Skipped 18658 bytes, use 'verb' option for the full verbose log.
cilium_1         | [...]
cilium_1         | r1
cilium_1         | 112: (15) if r2 == 0x0 goto pc+9
cilium_1         |  R0=inv R1=inv63 R2=inv R3=imm2 R6=ctx R7=imm0 R8=inv R9=inv48 R10=fp
cilium_1         | 113: (b7) r1 = 2
cilium_1         | 114: (63) *(u32 *)(r6 +48) = r1
cilium_1         | 115: (63) *(u32 *)(r6 +52) = r8
cilium_1         | 116: (bf) r1 = r6
cilium_1         | 117: (18) r2 = 0x14b75f00
cilium_1         | 119: (b7) r3 = 2
cilium_1         | 120: (85) call 12
cilium_1         | 121: safe
cilium_1         | 
cilium_1         | from 112 to 122: safe
cilium_1         | 
cilium_1         | from 1296 to 1388: R0=inv R6=ctx R7=imm0 R8=imm29898 R9=inv48 R10=fp
cilium_1         | 1388: (b7) r1 = 58
cilium_1         | 1389: (73) *(u8 *)(r10 -116) = r1
cilium_1         | 1390: (63) *(u32 *)(r10 -120) = r7
cilium_1         | 1391: (71) r1 = *(u8 *)(r10 -115)
cilium_1         | 1392: (bf) r2 = r1
cilium_1         | 1393: (47) r2 |= 2
cilium_1         | 1394: (73) *(u8 *)(r10 -115) = r2
cilium_1         | 1395: (47) r1 |= 14850
cilium_1         | 1396: (61) r2 = *(u32 *)(r6 +8)
cilium_1         | 1397: (55) if r2 != 0x0 goto pc+1
cilium_1         |  R0=inv R1=inv R2=inv R6=ctx R7=imm0 R8=imm29898 R9=inv48 R10=fp
cilium_1         | 1398: (61) r2 = *(u32 *)(r6 +68)
cilium_1         | 1399: (b7) r3 = 2
cilium_1         | 1400: (73) *(u8 *)(r10 -40) = r3
cilium_1         | 1401: (b7) r3 = 8
cilium_1         | 1402: (73) *(u8 *)(r10 -39) = r3
cilium_1         | 1403: (b7) r3 = 29898
cilium_1         | 1404: (6b) *(u16 *)(r10 -38) = r3
cilium_1         | 1405: (63) *(u32 *)(r10 -36) = r2
cilium_1         | 1406: (b7) r2 = 0
cilium_1         | 1407: (63) *(u32 *)(r10 -32) = r2
cilium_1         | 1408: (63) *(u32 *)(r10 -28) = r1
cilium_1         | 1409: (63) *(u32 *)(r10 -24) = r2
cilium_1         | 1410: (bf) r4 = r10
cilium_1         | 1411: (07) r4 += -40
cilium_1         | 1412: (bf) r1 = r6
cilium_1         | 1413: (18) r2 = 0x2c8ef300
cilium_1         | 1415: (18) r3 = 0xffffffff
cilium_1         | 1417: (b7) r5 = 20
cilium_1         | 1418: (85) call 25
cilium_1         | 1419: (bf) r2 = r10
cilium_1         | 1420: (07) r2 += -136
cilium_1         | 1421: (bf) r3 = r10
cilium_1         | 1422: (07) r3 += -80
cilium_1         | 1423: (18) r1 = 0x14b75780
cilium_1         | 1425: (b7) r4 = 0
cilium_1         | 1426: (85) call 2
cilium_1         | 1427: (67) r0 <<= 32
cilium_1         | 1428: (c7) r0 s>>= 63
cilium_1         | 1429: (bf) r8 = r0
cilium_1         | 1430: (57) r8 &= -155
cilium_1         | 1431: (65) if r0 s> 0xffffffff goto pc+148
cilium_1         |  R0=inv R6=ctx R7=imm0 R8=inv R9=inv48 R10=fp
cilium_1         | 1432: (05) goto pc-1330
cilium_1         | 103: safe
cilium_1         | 
cilium_1         | from 1431 to 1580: R0=inv R6=ctx R7=imm0 R8=inv R9=inv48 R10=fp
cilium_1         | 1580: (61) r1 = *(u32 *)(r10 -136)
cilium_1         | 1581: (55) if r1 != 0xdf0 goto pc+104
cilium_1         |  R0=inv R1=inv R6=ctx R7=imm0 R8=inv R9=inv48 R10=fp
cilium_1         | 1582: (61) r1 = *(u32 *)(r10 -132)
cilium_1         | 1583: (67) r1 <<= 32
cilium_1         | 1584: (77) r1 >>= 32
cilium_1         | 1585: (55) if r1 != 0x0 goto pc+100
cilium_1         |  R0=inv R1=inv32 R6=ctx R7=imm0 R8=inv R9=inv48 R10=fp
cilium_1         | 1586: (61) r1 = *(u32 *)(r10 -128)
cilium_1         | 1587: (bf) r2 = r1
cilium_1         | 1588: (57) r2 &= 65535
cilium_1         | 1589: (55) if r2 != 0xa8c0 goto pc+96
cilium_1         |  R0=inv R1=inv R2=inv48 R6=ctx R7=imm0 R8=inv R9=inv48 R10=fp
cilium_1         | 1590: (55) if r1 != 0xb22a8c0 goto pc+93
cilium_1         |  R0=inv R1=inv R2=inv48 R6=ctx R7=imm0 R8=inv R9=inv48 R10=fp
cilium_1         | 1591: (61) r1 = *(u32 *)(r6 +80)
cilium_1         | 1592: (61) r2 = *(u32 *)(r6 +76)
cilium_1         | 1593: (69) r3 = *(u16 *)(r10 -122)
cilium_1         | 1594: (b7) r4 = 65280
cilium_1         | 1595: (2d) if r4 > r3 goto pc+2
cilium_1         |  R0=inv R1=pkt_end R2=pkt(id=0,off=0,r=0) R3=inv48 R4=imm65280 R6=ctx R7=imm0 R8=inv R9=inv48 R10=fp
cilium_1         | 1596: (57) r3 &= 255
cilium_1         | 1597: (15) if r3 == 0xff goto pc+194
cilium_1         |  R0=inv R1=pkt_end R2=pkt(id=0,off=0,r=0) R3=inv56 R4=imm65280 R6=ctx R7=imm0 R8=inv R9=inv48 R10=fp
cilium_1         | 1598: (18) r8 = 0xffffff7a
cilium_1         | 1600: (bf) r3 = r2
cilium_1         | 1601: (07) r3 += 54
cilium_1         | 1602: (2d) if r3 > r1 goto pc-1500
cilium_1         |  R0=inv R1=pkt_end R2=pkt(id=0,off=0,r=54) R3=pkt(id=0,off=54,r=54) R4=imm65280 R6=ctx R7=imm0 R8=inv R9=inv48 R10=fp
cilium_1         | 1603: (b7) r1 = 0
cilium_1         | 1604: (63) *(u32 *)(r6 +56) = r1
cilium_1         | 1605: (71) r3 = *(u8 *)(r10 -116)
cilium_1         | 1606: (7b) *(u64 *)(r10 -160) = r3
cilium_1         | 1607: (61) r2 = *(u32 *)(r2 +50)
cilium_1         | 1608: (57) r2 &= -65536
cilium_1         | 1609: (dc) (u32) r2 endian (u32) r0
cilium_1         | 1610: (63) *(u32 *)(r10 -8) = r2
cilium_1         | 1611: (61) r3 = *(u32 *)(r6 +8)
cilium_1         | 1612: (55) if r3 != 0x0 goto pc+1
cilium_1         |  R0=inv R1=imm0 R2=inv R3=inv R4=imm65280 R6=ctx R7=imm0 R8=inv R9=inv48 R10=fp
cilium_1         | 1613: (61) r3 = *(u32 *)(r6 +68)
cilium_1         | 1614: (b7) r7 = 2
cilium_1         | 1615: (73) *(u8 *)(r10 -80) = r7
cilium_1         | 1616: (73) *(u8 *)(r10 -79) = r7
cilium_1         | 1617: (b7) r4 = 29898
cilium_1         | 1618: (6b) *(u16 *)(r10 -78) = r4
cilium_1         | 1619: (63) *(u32 *)(r10 -76) = r3
cilium_1         | 1620: (63) *(u32 *)(r10 -72) = r2
cilium_1         | 1621: (b7) r2 = 259
cilium_1         | 1622: (63) *(u32 *)(r10 -68) = r2
cilium_1         | 1623: (63) *(u32 *)(r10 -64) = r1
cilium_1         | 1624: (bf) r4 = r10
cilium_1         | 1625: (07) r4 += -80
cilium_1         | 1626: (bf) r1 = r6
cilium_1         | 1627: (18) r2 = 0x2c8ef300
cilium_1         | 1629: (18) r3 = 0xffffffff
cilium_1         | 1631: (b7) r5 = 20
cilium_1         | 1632: (85) call 25
cilium_1         | 1633: (bf) r2 = r10
cilium_1         | 1634: (07) r2 += -8
cilium_1         | 1635: (18) r1 = 0x2c8ef900
cilium_1         | 1637: (85) call 1
cilium_1         | 1638: (18) r8 = 0xffffff68
cilium_1         | 1640: (7b) *(u64 *)(r10 -152) = r0
cilium_1         | 1641: (15) if r0 == 0x0 goto pc-1539
cilium_1         |  R0=map_value(ks=4,vs=104) R6=ctx R7=imm2 R8=inv R9=inv48 R10=fp fp-152=map_value_or_null
cilium_1         | 1642: (79) r2 = *(u64 *)(r10 -152)
cilium_1         | 1643: (79) r1 = *(u64 *)(r2 +8)
cilium_1         | R2 invalid mem access 'map_value_or_null'
cilium_1         | 
cilium_1         | Error fetching program/map!
cilium_1         | Failed to retrieve (e)BPF data!
@aanm aanm added the kind/bug This is a bug in the Cilium logic. label Oct 20, 2016
@tgraf
Copy link
Member

tgraf commented Oct 20, 2016

The issue is that r1 gets pushed to the stack before the conditional jump. I think may recent upstream work should resolve this.

commit 57a09bf0a416700676e77102c28f9cfcb48267e0
Author: Thomas Graf <tgraf@suug.ch>
Date:   Tue Oct 18 19:51:19 2016 +0200

    bpf: Detect identical PTR_TO_MAP_VALUE_OR_NULL registers

    A BPF program is required to check the return register of a
    map_elem_lookup() call before accessing memory. The verifier keeps
    track of this by converting the type of the result register from
    PTR_TO_MAP_VALUE_OR_NULL to PTR_TO_MAP_VALUE after a conditional
    jump ensures safety. This check is currently exclusively performed
    for the result register 0.

    In the event the compiler reorders instructions, BPF_MOV64_REG
    instructions may be moved before the conditional jump which causes
    them to keep their type PTR_TO_MAP_VALUE_OR_NULL to which the
    verifier objects when the register is accessed:

    0: (b7) r1 = 10
    1: (7b) *(u64 *)(r10 -8) = r1
    2: (bf) r2 = r10
    3: (07) r2 += -8
    4: (18) r1 = 0x59c00000
    6: (85) call 1
    7: (bf) r4 = r0
    8: (15) if r0 == 0x0 goto pc+1
     R0=map_value(ks=8,vs=8) R4=map_value_or_null(ks=8,vs=8) R10=fp
    9: (7a) *(u64 *)(r4 +0) = 0
    R4 invalid mem access 'map_value_or_null'

    This commit extends the verifier to keep track of all identical
    PTR_TO_MAP_VALUE_OR_NULL registers after a map_elem_lookup() by
    assigning them an ID and then marking them all when the conditional
    jump is observed.

    Signed-off-by: Thomas Graf <tgraf@suug.ch>
    Reviewed-by: Josef Bacik <jbacik@fb.com>
    Acked-by: Daniel Borkmann <daniel@iogearbox.net>
    Acked-by: Alexei Starovoitov <ast@kernel.org>
    Signed-off-by: David S. Miller <davem@davemloft.net>

@aanm aanm added the postponed label Oct 20, 2016
@tgraf
Copy link
Member

tgraf commented Mar 6, 2017

Fixed in kernel commit 57a09bf0a416700676e77102c28f9cfcb48267e0

@tgraf tgraf closed this as completed Mar 6, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug This is a bug in the Cilium logic.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tgraf @aanm