mirrored from git://xenbits.xen.org/xen.git
-
Notifications
You must be signed in to change notification settings - Fork 327
/
grant-tables.txt
383 lines (281 loc) · 14 KB
/
grant-tables.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
********************************************************************************
A Rough Introduction to Using Grant Tables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Christopher Clark, March, 2005.
Grant tables are a mechanism for sharing and transferring frames between
domains, without requiring the participating domains to be privileged.
The first mode of use allows domA to grant domB access to a specific frame,
whilst retaining ownership. The block front driver uses this to grant memory
access to the block back driver, so that it may read or write as requested.
1. domA creates a grant access reference, and transmits the ref id to domB.
2. domB uses the reference to map the granted frame.
3. domB performs the memory access.
4. domB unmaps the granted frame.
5. domA removes its grant.
The second mode allows domA to accept a transfer of ownership of a frame from
domB. The net front and back driver will use this for packet tx/rx. This
mechanism is still being implemented, though the xen<->guest interface design
is complete.
1. domA creates an accept transfer grant reference, and transmits it to domB.
2. domB uses the ref to hand over a frame it owns.
3. domA accepts the transfer
4. domA clears the used reference.
********************************************************************************
Data structures
~~~~~~~~~~~~~~~
The following data structures are used by Xen and the guests to implement
grant tables:
1. Shared grant entries
2. Active grant entries
3. Map tracking
These are not the users primary interface to grant tables, but are discussed
because an understanding of how they work may be useful. Each of these is a
finite resource.
Shared grant entries
~~~~~~~~~~~~~~~~~~~~
A set of pages are shared between Xen and a guest, holding the shared grant
entries. The guest writes into these entries to create grant references. The
index of the entry is transmitted to the remote domain: this is the
reference used to activate an entry. Xen will write into a shared entry to
indicate to a guest that its grant is in use.
sha->domid : remote domain being granted rights
sha->frame : machine frame being granted
sha->flags : allow access, allow transfer, remote is reading/writing, etc.
Active grant entries
~~~~~~~~~~~~~~~~~~~~
Xen maintains a set of private frames per domain, holding the active grant
entries for safety, and to reference count mappings.
act->domid : remote domain being granted rights
act->frame : machine frame being granted
act->pin : used to hold reference counts
act->lock : spinlock used to serialize access to active entry state
Map tracking
~~~~~~~~~~~~
Every time a frame is mapped, a map track entry is stored in the metadata of
the mapping domain. The index of this entry is returned from the map call,
and is used to unmap the frame. Map track entries are also searched whenever a
page table entry containing a foreign frame number is overwritten: the first
matching map track entry is then removed, as if unmap had been invoked.
These are not used by the transfer mechanism.
map->domid : owner of the mapped frame
map->ref : grant reference
map->flags : ro/rw, mapped for host or device access
********************************************************************************
Locking
~~~~~~~
Xen uses several locks to serialize access to the internal grant table state.
grant_table->lock : rwlock used to prevent readers from accessing
inconsistent grant table state such as current
version, partially initialized active table pages,
etc.
grant_table->maptrack_lock : spinlock used to protect the maptrack limit
v->maptrack_freelist_lock : spinlock used to protect the maptrack free list
active_grant_entry->lock : spinlock used to serialize modifications to
active entries
The primary lock for the grant table is a read/write spinlock. All
functions that access members of struct grant_table must acquire a
read lock around critical sections. Any modification to the members
of struct grant_table (e.g., nr_status_frames, nr_grant_frames,
active frames, etc.) must only be made if the write lock is
held. These elements are read-mostly, and read critical sections can
be large, which makes a rwlock a good choice.
The maptrack free list is protected by its own spinlock. The maptrack
lock may be locked while holding the grant table lock.
The maptrack_freelist_lock is an innermost lock. It may be locked
while holding other locks, but no other locks may be acquired within
it.
Active entries are obtained by calling active_entry_acquire(gt, ref).
This function returns a pointer to the active entry after locking its
spinlock. The caller must hold the grant table read lock before
calling active_entry_acquire(). This is because the grant table can
be dynamically extended via gnttab_grow_table() while a domain is
running and must be fully initialized. Once all access to the active
entry is complete, release the lock by calling active_entry_release(act).
Summary of rules for locking:
active_entry_acquire() and active_entry_release() can only be
called when holding the relevant grant table's read lock. I.e.:
read_lock(>->lock);
act = active_entry_acquire(gt, ref);
...
active_entry_release(act);
read_unlock(>->lock);
Active entries cannot be acquired while holding the maptrack lock.
Multiple active entries can be acquired while holding the grant table
_write_ lock.
Maptrack entries are protected by the corresponding active entry
lock. As an exception, new maptrack entries may be populated without
holding the lock, provided the flags field is written last. This
requires any maptrack entry user validates the flags field as
non-zero first.
********************************************************************************
Granting a foreign domain access to frames
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
domA [frame]--> domB
domA: #include <asm-xen/gnttab.h>
grant_ref_t gref[BATCH_SIZE];
for ( i = 0; i < BATCH_SIZE; i++ )
gref[i] = gnttab_grant_foreign_access( domBid, mfn, (readonly ? 1 : 0) );
.. gref is then somehow transmitted to domB for use.
Mapping foreign frames
~~~~~~~~~~~~~~~~~~~~~~
domB: #include <asm-xen/hypervisor.h>
unsigned long mmap_vstart;
gnttab_op_t aop[BATCH_SIZE];
grant_ref_t mapped_handle[BATCH_SIZE];
if ( (mmap_vstart = allocate_empty_lowmem_region(BATCH_SIZE)) == 0 )
BUG();
for ( i = 0; i < BATCH_SIZE; i++ )
{
aop[i].u.map_grant_ref.host_virt_addr =
mmap_vstart + (i * PAGE_SIZE);
aop[i].u.map_grant_ref.dom = domAid;
aop[i].u.map_grant_ref.ref = gref[i];
aop[i].u.map_grant_ref.flags = ( GNTMAP_host_map | GNTMAP_readonly );
}
if ( unlikely(HYPERVISOR_grant_table_op(
GNTTABOP_map_grant_ref, aop, BATCH_SIZE)))
BUG();
for ( i = 0; i < BATCH_SIZE; i++ )
{
if ( unlikely(aop[i].u.map_grant_ref.handle < 0) )
{
tidyup_all(aop, i);
goto panic;
}
phys_to_machine_mapping[__pa(mmap_vstart + (i * PAGE_SIZE))>>PAGE_SHIFT] =
FOREIGN_FRAME(aop[i].u.map_grant_ref.dev_bus_addr);
mapped_handle[i] = aop[i].u.map_grant_ref.handle;
}
Unmapping foreign frames
~~~~~~~~~~~~~~~~~~~~~~~~
domB:
for ( i = 0; i < BATCH_SIZE; i++ )
{
aop[i].u.unmap_grant_ref.host_virt_addr = mmap_vstart + (i * PAGE_SIZE);
aop[i].u.unmap_grant_ref.dev_bus_addr = 0;
aop[i].u.unmap_grant_ref.handle = mapped_handle[i];
}
if ( unlikely(HYPERVISOR_grant_table_op(
GNTTABOP_unmap_grant_ref, aop, BATCH_SIZE)))
BUG();
Ending foreign access
~~~~~~~~~~~~~~~~~~~~~
Note that this only prevents further mappings; it does _not_ revoke access.
Should _only_ be used when the remote domain has unmapped the frame.
gnttab_query_foreign_access( gref ) will indicate the state of any mapping.
domA:
if ( gnttab_query_foreign_access( gref[i] ) == 0 )
gnttab_end_foreign_access( gref[i], readonly );
TODO: readonly yet to be implemented.
********************************************************************************
Transferring ownership of a frame to another domain
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ XXX: Transfer mechanism is alpha-calibre code, untested, use at own risk XXX ]
[ XXX: show use of batch operations below, rather than single frame XXX ]
[ XXX: linux internal interface could/should be wrapped to be tidier XXX ]
Prepare to accept a frame from a foreign domain
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
domA:
if ( (p = alloc_page(GFP_HIGHUSER)) == NULL )
{
printk("Cannot alloc a frame to surrender\n");
break;
}
pfn = p - mem_map;
mfn = phys_to_machine_mapping[pfn];
if ( !PageHighMem(p) )
{
v = phys_to_virt(pfn << PAGE_SHIFT);
scrub_pages(v, 1);
queue_l1_entry_update(get_ptep((unsigned long)v), 0);
}
/* Ensure that ballooned highmem pages don't have cached mappings. */
kmap_flush_unused();
/* Flush updates through and flush the TLB. */
xen_tlb_flush();
phys_to_machine_mapping[pfn] = INVALID_P2M_ENTRY;
if ( HYPERVISOR_dom_mem_op(
MEMOP_decrease_reservation, &mfn, 1, 0) != 1 )
{
printk("MEMOP_decrease_reservation failed\n");
/* er... ok. free the page then */
__free_page(p);
break;
}
accepting_pfn = pfn;
ref = gnttab_grant_foreign_transfer( (domid_t) args.arg[0], pfn );
printk("Accepting dom %lu frame at ref (%d)\n", args.arg[0], ref);
Transfer a frame to a foreign domain
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
domB:
mmu_update_t update;
domid_t domid;
grant_ref_t gref;
unsigned long pfn, mfn, *v;
struct page *transfer_page = 0;
/* alloc a page and grant access.
* alloc page returns a page struct. */
if ( (transfer_page = alloc_page(GFP_HIGHUSER)) == NULL )
return -ENOMEM;
pfn = transfer_page - mem_map;
mfn = phys_to_machine_mapping[pfn];
/* need to remove all references to this page */
if ( !PageHighMem(transfer_page) )
{
v = phys_to_virt(pfn << PAGE_SHIFT);
scrub_pages(v, 1);
sprintf((char *)v, "This page (%lx) was transferred.\n", mfn);
queue_l1_entry_update(get_ptep((unsigned long)v), 0);
}
#ifdef CONFIG_XEN_SCRUB_PAGES
else
{
v = kmap(transfer_page);
scrub_pages(v, 1);
sprintf((char *)v, "This page (%lx) was transferred.\n", mfn);
kunmap(transfer_page);
}
#endif
/* Delete any cached kmappings */
kmap_flush_unused();
/* Flush updates through and flush the TLB */
xen_tlb_flush();
/* invalidate in P2M */
phys_to_machine_mapping[pfn] = INVALID_P2M_ENTRY;
domid = (domid_t)args.arg[0];
gref = (grant_ref_t)args.arg[1];
update.ptr = MMU_EXTENDED_COMMAND;
update.ptr |= ((gref & 0x00FF) << 2);
update.ptr |= mfn << PAGE_SHIFT;
update.val = MMUEXT_TRANSFER_PAGE;
update.val |= (domid << 16);
update.val |= (gref & 0xFF00);
ret = HYPERVISOR_mmu_update(&update, 1, NULL);
Map a transferred frame
~~~~~~~~~~~~~~~~~~~~~~~
TODO:
Clear the used transfer reference
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
TODO:
********************************************************************************
Using a private reserve of grant references
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Where it is known in advance how many grant references are required, and
failure to allocate them on demand would cause difficulty, a batch can be
allocated and held in a private reserve.
To reserve a private batch:
/* housekeeping data - treat as opaque: */
grant_ref_t gref_head, gref_terminal;
if ( 0 > gnttab_alloc_grant_references( number_to_reserve,
&gref_head, &gref_terminal ))
return -ENOSPC;
To release a batch back to the shared pool:
gnttab_free_grant_references( number_reserved, gref_head );
To claim a reserved reference:
ref = gnttab_claim_grant_reference( &gref_head, gref_terminal );
To release a claimed reference back to the reserve pool:
gnttab_release_grant_reference( &gref_head, gref );
To use a claimed reference to grant access, use these alternative functions
that take an additional parameter of the grant reference to use:
gnttab_grant_foreign_access_ref
gnttab_grant_foreign_transfer_ref