tokenizer_test.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "879abb0b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, DataCollatorWithPadding\n",
    "\n",
    "checkpoint = 'bert-base-cased'\n",
    "tokenizer = AutoTokenizer.from_pretrained(checkpoint)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b49d38cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"When a man who doesn't have Alzheimer's can't remember how many films he's made, he probably is the world's most prolific director after all. That man is Jesus Franco, the king of so-called 'eurotrash'. His 1980 flick Devil Hunter is as rushed, opaque, stupid, lazy and exploitative in the truest sense of the word (the film's title is misleading, for starters) as any other Franco film I've seen. That makes it sound pretty awful, and it is... Yet Franco does have some kind of inimitable sensibility, a generous way with the baldly outrageous, with nudity and sleaze and violence, and even with his stupid cheap editing which tries to pave over the extreme haste with which all his films were made. The mix of all these elements causes you to ride his films out, even while you're mostly waiting for them to end because they're so very tedious.<br /><br />Devil Hunter is nigh on incomprehensible for the first half an hour. The kidnap by strangers of a white woman who seems to be a model or film star is intercut with a bunch of native action in South America. There's lots of naked writhing, dancing, and endless repeated zoom-ins on an ugly totem pole. You need to get used to the repetitive zoom-ins and the technique of cutting back to the same shot about three times in a row right away, as these are Franco's main methods of extending a film out to feature length.<br /><br />The monster who looks like the totem pole is actually kind of scary. He has raw bug eyes and his presence is always signalled on the soundtrack by cacophonous groaning, apparently recorded in an echo chamber. Early in the piece he chews on a native lady strapped to a tree, and it's hard to know what really happens here but I think he ate her stomach (or her genitals, sweet Jesus!).<br /><br />Anyway, the adventure begins properly when a studly guy and his freaked out Vietnam vet pal are sent to the island to recover the white girl from the kidnappers. The flakey guy has an accent which, as dubbed, is half Brooklyn-American, half English-Liverpudlian and all retarded. All of the dialogue and dubbing is ridiculous and laughable, making for another layer of the film which can somehow hold your interest.<br /><br />Not too much really happens from here on in, and it happens pretty sluggishly, studded with the odd bit of outrage like a rape. The nebulous action is fleshed out (haha!) by acres of 360 degree nudity from the natives and the two female leads, and even from the monster himself. That he walks around with his penis exposed makes wrestling him an unappetising prospect for the tough guy hero, but it's gotta be done at some point, and it's nice to note that the director will show anyone's genitals on camera.<br /><br />The best feature of Devil Hunter is the location filming. Franco can be extremely cheap with the structural and story aspects of film-making, but he doesn't muck around with sets. You get real islands, jungles, helicopters and mountains, all in widescreen. This is something that is really cool to experience in these days of crappy CGI sets and backdrops ad nauseam.<br /><br />Ultimately, issues of recommendation where this film is concerned seem moot. If you're trying to see all the Video Nasties, you will have to watch this at some point, and you'll be made as restless as I was. If you like Franco, you'll watch this anyway. If you fall into neither of the above categories, the odds are you'll never come across this film. Copies of it aren't just lying around, and I could hardly recommend the seeking out of it. It's Franco. Lazy, crazy Franco.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "71e111bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = tokenizer(text, max_length=100, truncation=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "51fe020d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': [101, 1332, 170, 1299, 1150, 2144, 112, 189, 1138, 24278, 112, 188, 1169, 112, 189, 2676, 1293, 1242, 2441, 1119, 112, 188, 1189, 117, 1119, 1930, 1110, 1103, 1362, 112, 188, 1211, 13127, 1900, 1170, 1155, 119, 1337, 1299, 1110, 3766, 9063, 117, 1103, 2226, 1104, 1177, 118, 1270, 112, 27772, 25352, 1324, 112, 119, 1230, 2253, 22302, 8201, 4242, 1110, 1112, 6169, 117, 11769, 23911, 117, 4736, 117, 16688, 1105, 19685, 5838, 1107, 1103, 2276, 2050, 2305, 1104, 1103, 1937, 113, 1103, 1273, 112, 188, 1641, 1110, 25175, 117, 1111, 12302, 1116, 114, 1112, 1251, 1168, 9063, 1273, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "507a1124",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(result[\"input_ids\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eec656ea",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}