New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accessing GPIO from TA in Raspberry Pi 3 #1496

Closed
utkarshagrawalwsu opened this Issue Apr 24, 2017 · 13 comments

Comments

Projects
None yet
3 participants
@utkarshagrawalwsu

utkarshagrawalwsu commented Apr 24, 2017

Hello,

I have been trying to access the Raspberry Pi3 GPIO from TA. What would be a secure manner to do so?

I tried using the sysfs method mentioned in the following link:
http://elinux.org/RPi_GPIO_Code_Samples#sysfs
But it requires the use of C file operations. Does OP-TEE support file operations in TA?

Thank You

@jforissier

This comment has been minimized.

Contributor

jforissier commented Apr 24, 2017

I have been trying to access the Raspberry Pi3 GPIO from TA. What would be a secure manner to do so?

Do the GPIO stuff in a pseudo-TA (EL1), where you can access the hardware registers. The pseudo-TA can then be invoked from a client app or a user TA.

I tried using the sysfs method mentioned in the following link:
http://elinux.org/RPi_GPIO_Code_Samples#sysfs
But it requires the use of C file operations. Does OP-TEE support file operations in TA?

sysfs is a Linux (normal world) interface, so it obviously cannot work.

@utkarshagrawalwsu

This comment has been minimized.

utkarshagrawalwsu commented Apr 25, 2017

Hello @jforissier

Thank you for your response.

I don't have any experience to access a register directly from C. Though, let me try it and get back to you with it. Also, the link below(and some other search results) mentions about accessing GPIO through direct register access. It still uses file operation as open("/dev/mem", O_RDWR|O_SYNC). I wanted to know if it is still possible to access the GPIO this way, as there is no definition for open() function.
http://elinux.org/RPi_GPIO_Code_Samples#Direct_register_access

I was under the impression that the TA's can access resources in the normal world. Is there any way to do so from the TA? Or is there any other way to access the GPIO from the dynamic TA?

I appreciate your help with it.

@jforissier

This comment has been minimized.

Contributor

jforissier commented Apr 26, 2017

Also, the link below(and some other search results) mentions about accessing GPIO through direct register access. It still uses file operation as open("/dev/mem", O_RDWR|O_SYNC). I wanted to know if it is still possible to access the GPIO this way, as there is no definition for open() function.
http://elinux.org/RPi_GPIO_Code_Samples#Direct_register_access

No. A "dynamic" or "user" TA is not a regular Linux application. The runtime environment is very different (although some standard libc functions such as assert() or malloc() are available). In a TA, you are supposed to use the GlobalPlatform API only. There is no open() function. The normal world devices (/dev/*) are not accessible. You can't access the hardware (GPIO...) directly from a TA, because it is running in user mode (SEL0) and the hardware registers are not mapped to the TA virtual address space.

A pseudo-TA, however, allows hardware access. It is a bit of code that is statically linked with OP-TEE, runs in secure "kernel" mode (SEL1) and therefore may access physical addresses (hence the memory-mapped hardware registers). See also https://github.com/OP-TEE/optee_os/blob/master/documentation/optee_design.md#12-trusted-applications. For examples of pseudo-TAs, see https://github.com/OP-TEE/optee_os/tree/master/core/arch/arm/pta. There you could create an app that would access the GPIO as you wish, possibly similar to what is done in https://github.com/OP-TEE/optee_os/blob/master/core/arch/arm/plat-hikey/spi_test.c, and exposes a service that can be invoked from a client application in normal world or a dynamic TA.

Note that another option to control the hardware from a dynamic TA is to add a system call to the OP-TEE kernel, and invoke it from the TA. But it is not recommended because it involves extending the kernel/user interface (syscall, syscall wrapper, new function in libutee...).

I was under the impression that the TA's can access resources in the normal world. Is there any way to do so from the TA? Or is there any other way to access the GPIO from the dynamic TA?

No. Only indirect access is possible: as I said above, your dynamic TA may call a pseudo-TA specifically designed for that purpose, for instance.

@utkarshagrawalwsu

This comment has been minimized.

utkarshagrawalwsu commented Apr 26, 2017

@jforissier

Thank you for clarifying about the user and pseudo TA. I wrote my first pseudo TA(I have been working with user TA only since past few months), and was able to successfully expose a dummy service to the user TA. That's one step.

I looked at the spi_test.c, and honestly I could not understand much, as I do not have much low level programming experience. So please bear with me if I pose any dumb question.

I have been looking at some guides to directly access GPIO by writing your own driver:
http://sysprogs.com/VisualKernel/tutorials/raspberry/leddriver/
In the above instruction, at step 4, they do the following:
s_pGpioRegisters = (struct GpioRegisters *)__io_address(GPIO_BASE);
Where do I find the __io_address() function?

Right now all I am trying is to light up a LED from the pseudo TA.

Thank you again for your assistance.

@jforissier

This comment has been minimized.

Contributor

jforissier commented Apr 26, 2017

@vchong I think this is for you ;) Would you please share your experience re. HiKey GPIO programming in OP-TEE?

@vchong

This comment has been minimized.

Contributor

vchong commented Apr 27, 2017

@utkarshagrawalwsu If I did my research correctly, rpi3 uses a broadcom bcm2837 soc (https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2837). It seems like there's no spec (hardware datasheet) for the 2837, but the url says that its underlying architecture is identical to bcm2836 from rpi2 model b, which is identical to bcm2835 from rpi, which brings us to https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2835/BCM2835-ARM-Peripherals.pdf. In section 6, you can see all the register addresses and values you would need to write to these registers in order to control the gpio functionalities.

I'm not familiar with the bcm283x, but basically, you would need to understand how these register works, e.g. if you want to configure gpio pin number 2 to be an output pin, what value do you need to write to configure a gpio pin as output (vs input), and to which register address you need to write that value to, i.e. which bit in which register address refers to pin 2. Then you will do this in your pseudo TA using write32() and friends functions from io.h in optee_os. See pl061_gpio.c for examples of how this is done in Hikey.

I actually couldn't find where the gpio driver for 2835 is in https://github.com/raspberrypi/linux :(, but you can find useful reference code in http://www.airspayce.com/mikem/bcm2835/, which is a straightforward (relatively simple) implementation of the above BCM2835-ARM-Peripherals.pdf, and port the necessary functions to your pseudo TA. The code there seems to be updated to rpi2 only so I'm not sure if it's a 1:1 match to rpi3. Most probably not, so you'll have to adjust the addresses, pin numbers, etc properly to adapt it to rpi3 (e.g. in bcm2835.h), but the basic functions and framework are there. There are a lot bit manipulations (shifting, masking, etc) in the code, common to most low level programming, which is why, again, you really have to understand how the register works first before you can understand the code. It will involve a fair amount of research and reading the specs and code, especially more so if you're new to this, so please take things one step at a time.

@vchong

This comment has been minimized.

Contributor

vchong commented Apr 27, 2017

@utkarshagrawalwsu Once you're done with the above, to light up an LED from a GPIO software perspective, you would normally:

  1. Find out the gpio pin number you need
  2. Configure the function for that pin number as gpio. Usually this is the default. If so, you don't need to do this step. Other functions are like spi, i2c, uart, etc.
  3. Disable interrupt on the pin if necessary, but maybe optional in most cases I think.
  4. Configure the gpio pin as output. Usually the default is input.
  5. Set the gpio pin level to high to light up the LED connected, or low to turn it off.

Steps 3-5 for the hikey platform are done in spi_test.c lines 51, 52 and 62 respectively, which was why you were referred to it earlier, but these are done through driver function calls which you might not want to worry about atm. For now, you would just use functions in io.h to write the appropriate values to the appropriate registers directly.

@jforissier

This comment has been minimized.

Contributor

jforissier commented Apr 27, 2017

@vchong thanks for doing all the research work, I'm sorry I am just realizing the OP is using rpi3 and not HiKey! Doh! ;)

@vchong

This comment has been minimized.

Contributor

vchong commented Apr 27, 2017

@jforissier yep, np. ;) It was good info to find out and know.

@utkarshagrawalwsu

This comment has been minimized.

utkarshagrawalwsu commented Apr 28, 2017

@jforissier Thank you for introducing @vchong to the issue

Hey @vchong !
Thank you for your elaborate reply. Your research aligns with what I have found too.

I have come up with the following code to set GPIO pin 18 as output pin, and give it a high value.

// Set the GPIO Pin 18 in output mode
static void SetAsOutput() {
    // Because we are accessing GPIO 18,
    // we need base address for GPFSEL1
    vaddr_t base_addr_GPFSEL1 = 0x7E200004;

    // Bits 26-24 need to be cleared
    // 00 000 111 000 000 000 000 000 000 000 000
    uint32_t clearMask = 0x7000000;

    // Bits 26-24 need to be set to output mode 001
    // 00 000 001 000 000 000 000 000 000 000 000
    uint32_t enableWrite = 0x1000000;

    uint32_t data;

    data = read32(base_addr_GPFSEL1);
    data = data & ~clearMask;
    data = data | enableWrite;

    write32(data, base_addr_GPFSEL1);
}

// Set the GPIO Pin 18 with a high value
static void SetGPIOOutputValue() {
    // Because we are accessing GPIO 18,
    // we need base address for GPSET0
    vaddr_t base_addr_GPSET0 = 0x7E20001C;

    // Bit 18 needs to be cleared
    uint32_t clearMask = 0x40000;

    // Bit 18 needs to be set to 1
    uint32_t makeHigh = 0x40000;

    uint32_t data;

    data = read32(base_addr);
    data = data & ~clearMask;
    data = data | makeHigh;

    write32(data, base_addr_GPSET0);
}

The above code doesn't work though. The screen freezes for sometime and then the outputs as follows:
mmc0: timeout waiting for hardware interrupt

Just as a note, I invoke two commands in the pseudo TA from the host. The first command I invoke is similar to the hello world program and it is just to make sure that the pseudo TA calls are successful. The second command actually tries to play around with the GPIO stuff.

What base address should I use for the GPIO? Should it be the virtual address(0x3F000000 + 0x200000) or the physical address(0x7E200000)?

@vchong

This comment has been minimized.

Contributor

vchong commented Apr 28, 2017

@utkarshagrawalwsu You're welcome! Sounds like you made good progress! :) I only browsed through the spec so can't say for sure if the bits selection is right or not (although it looks ok from a quick glance). You'll have to trial and error until you find the right combinations.

Off the top of my head, several things to look at.

mmc0: timeout waiting for hardware interrupt

Not sure if this is related, but is it possible to disable interrupt on the pin? I don't see any interrupt related registers for gpio, but they might be located elsewhere. There are mentions of gpio_int in the spec, but I didn't look further.

What base address should I use for the GPIO? Should it be the virtual address(0x3F000000 + 0x200000) or the physical address(0x7E200000)?

I wish the article explained how Dom managed to find out 0x3F200000 for rpi3, but it's not the va. It is actually the pa. 0x7E200000 is the bus address. Read Section 1, especially 1.2 and 1.3, of the spec too, not just the gpio section. The address mapping is multiple layered and hard to figure out (at least for me). I think the pa (0x3F200000) should be used, since there's no dma engines involved, but I might be wrong.

If using pa, and it doesn't work (you might get a core dump), you might also to consider trying something like:
register_phys_mem(MEM_AREA_IO_SEC, 0x3F200000, CORE_MMU_DEVICE_SIZE);

and translate the base address from pa to va before using them in your functions. Something like:

vaddr_t nsec_periph_base(paddr_t pa)
{
	if (cpu_mmu_enabled())
		return (vaddr_t)phys_to_virt(pa, MEM_AREA_IO_NSEC);
	return (vaddr_t)pa;
}

vaddr_t base_addr_GPFSEL0 = nsec_periph_base(0x3F200004);
vaddr_t base_addr_GPSET0 = nsec_periph_base(0x3F20001C);

Again, mostly guesswork here so you'll have to play around a bit to see what works.

@utkarshagrawalwsu

This comment has been minimized.

utkarshagrawalwsu commented May 1, 2017

@vchong

Success! The LED blinks!

The nsec_periph_base() was the key. Including register_phys_mem() function did not seem to work. I couldn't even build the code with it. I get the following error:

/home/wsu/PiOptee1/build/../toolchains/aarch64/bin/aarch64-linux-gnu-ld: BFD (Linaro_Binutils-2016.11) 2.27.0.20161019 assertion fail /home/tcwg-buildslave/workspace/tcwg-make-release/label/docker-trusty-amd64-tcwg-build/target/aarch64-linux-gnu/snapshots/binutils-gdb.git~linaro_binutils-2_27-branch/bfd/elflink.c:8380
core/arch/arm/kernel/link.mk:90: recipe for target 'out/arm/core/init.o' failed

I do not have /home/tcwg-buildslave folder on my system.

It still bothers me where 0x3F200000 comes from. When the system boots up, I do see something like this:

pinctrl-bcm2835 3f200000.gpio: Starting probe
pinctrl-bcm2835 3f200000.gpio: Probe successful

Well, the working code is as follows. Might help someone else. Two commands GPIO_ON and GPIO_OFF are defined, and the GPIO Pin 18 is controlled.

#define GPIO_ON             0
#define GPIO_OFF            1

#define RPI3_PERI_BASE      0x3F000000
#define GPIO_BASE           (RPI3_PERI_BASE + 0x200000)

typedef enum {
    GPIO_FSEL_INPT  = 0x00,
    GPIO_FSEL_OUTP  = 0x01
} FunctionSelect_GPFSEL;

typedef enum {
    GPIO_LOW     =  0x0,
    GPIO_HIGH    =  0x1
} GPIO_Value;

struct GpioRegister {
    uint32_t GPFSEL[6];
    uint32_t Reserved1;
    uint32_t GPSET[2];
    uint32_t Reserved2;
    uint32_t GPCLR[2];
};

struct GpioRegister *gpioRegister;

static vaddr_t nsec_periph_base(paddr_t pa) {
    if (cpu_mmu_enabled()) {
        return (vaddr_t)phys_to_virt(pa, MEM_AREA_IO_NSEC);
    }
    return (vaddr_t)pa;
}

static vaddr_t get_base_address_GPFSEL(uint8_t index) {
    return nsec_periph_base((paddr_t)&(gpioRegister->GPFSEL[index]));
}

static vaddr_t get_base_address_GPSET(uint8_t index) {
    return nsec_periph_base((paddr_t)&(gpioRegister->GPSET[index]));
}

static vaddr_t get_base_address_GPCLR(uint8_t index) {
    return nsec_periph_base((paddr_t)&(gpioRegister->GPCLR[index]));
}

static void set_gpio_pin_function(uint8_t pinNumber, FunctionSelect_GPFSEL functionCode) {
    uint32_t index_GPFSEL = pinNumber / 10;
    uint32_t bit_GPFSEL = (pinNumber % 10) * 3;

    vaddr_t base_addr_GPFSEL = get_base_address_GPFSEL(index_GPFSEL);
    uint32_t mask = 0x7 << bit_GPFSEL;

    uint32_t data;

    data = read32(base_addr_GPFSEL);

    data = data & ~mask;
    data = data | (functionCode << bit_GPFSEL);
    write32(data, base_addr_GPFSEL);
}

static void set_gpio_pin_value(uint8_t pinNumber, GPIO_Value value) {
    uint32_t index_register = pinNumber / 32;
    uint32_t bit_register = pinNumber % 32;

    vaddr_t base_addr = (value == GPIO_LOW) ? get_base_address_GPCLR(index_register) : get_base_address_GPSET(index_register);
    uint32_t mask = 0x1 << bit_register;

    uint32_t data;

    data = read32(base_addr);

    data = data & ~mask;
    data = data | (0x1 << bit_register);
    write32(data, base_addr);
}

static void initializeGpio(void) {
    gpioRegister = (struct GpioRegister *)(GPIO_BASE);
}

static TEE_Result testGpioOn(void) {
    static const int LedGpioPin = 18;

    initializeGpio();

    set_gpio_pin_function(LedGpioPin, GPIO_FSEL_OUTP);
    set_gpio_pin_value(LedGpioPin, GPIO_HIGH);
    
    return TEE_SUCCESS;
}

static TEE_Result testGpioOff(void) {
    static const int LedGpioPin = 18;

    initializeGpio();

    set_gpio_pin_function(LedGpioPin, GPIO_FSEL_OUTP);
    set_gpio_pin_value(LedGpioPin, GPIO_LOW);
    
    return TEE_SUCCESS;
}

static TEE_Result invoke_command(
	void *psess __unused,
	uint32_t cmd,
	uint32_t ptypes, TEE_Param params[TEE_NUM_PARAMS]
) {
	(void)ptypes;
	(void)params;

	switch (cmd) {
		case GPIO_ON:
			return testGpioOn();
		case GPIO_OFF:
			return testGpioOff();
		default:
			break;
	}
	return TEE_ERROR_BAD_PARAMETERS;
}

Thank you so much for your patience and your help @vchong and @jforissier

@vchong

This comment has been minimized.

Contributor

vchong commented May 2, 2017

Great! You're welcome and thanks for sharing!

register_phys_mem() function did not seem to work.

I think this is because it's already done in main.c for rpi3. The registration of CONSOLE_UART_BASE is probably large enough to cover that of the gpio as well, so you don't have to do it again.

Now, if you want to go a step further and do things the 'proper' way, you would write a gpio driver for the bcm2835, and control the gpio in your pseudo TA via the driver instead of directly modifying the registers. The contribution would definitely be appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment