TL/DR:
- Labels are currently a work in progress (WIP) for 1.0.
- Labels can be thought of as tags / attributes, not as a single classification with the Synopsis model. ie: A file may have multiple classifications associated with it assuming high enough probability. See Coco Attributes for an example of an attribute / multiclass labelling model.
- Labels use a reverse dns nomenclature from category to specific concepts and sub concepts. For example:
A single frame might be tagged with:
color.saturation.neutralcolor.theory.analagouscompositon.texture.smoothshot.framing.closeupshot.subject.personshot.subject.person.faceshot.location.indoorshot.location.roomshot.location.room.livingroom
This allows us to build a pseudo knowledge graph and not pollute name spaces of labels. Currently labels are broken up into a few main categories their individual named concepts.
Note that inorder to label a large data set, we train image classifiers (not multi label) on sub sets of concepts the proposed concepts that are mutually exclusive conceptually.
ColorCompositionShot
Color psychology and color theory
How do we describe the overall saturation of the image
color.saturation.desaturatedcolor.saturation.mutedcolor.saturation.neutralcolor.saturation.pastelcolor.saturation.saturated
How do we describe the color relationship in the image
color.theory.na(not applicable)color.theory.analagouscolor.theory.complementarycolor.theory.monochromatic
How do we describe the color temperature and tone in the image
color.tones.na(not applicable)color.tones.blackwhitecolor.tones.coolcolor.tones.warm
Does the image contain a luma or chroma key?
color.key.na(not applicable)color.key.lumacolor.key.greencolor.key.blue
How do we describe the human readable dominant colors - using most commonly used color names
color.dominant.na(not applicable)color.dominant.whitecolor.dominant.greycolor.dominant.blackcolor.dominant.redcolor.dominant.orangecolor.dominant.yellowcolor.dominant.limecolor.dominant.greencolor.dominant.cyancolor.dominant.bluecolor.dominant.purplecolor.dominant.magentacolor.dominant.brown
Taken from Gestalt Theory and graphic design / layout systems thinking
Does the image strogly consist of a pattern, and if so, what type of pattern?
composition.pattern.na(not applicable)composition.pattern.fractalcomposition.pattern.reflectcomposition.pattern.spiralcomposition.pattern.spotcomposition.pattern.stripecomposition.pattern.tile
How can we descibe the spatial layout and composition of the image?
composition.spatial.perspectivecomposition.spatial.orthographiccomposition.spatial.isometriccomposition.spatial.opencomposition.spatial.closedcomposition.spatial.densecomposition.spatial.sparsecomposition.spatial.horizoncomposition.spatial.verticalitycomposition.spatial.horitzontalitycomposition.spatial.diagonalitycomposition.spatial.ruleofthirdscomposition.spatial.negativespacecomposition.spatial.symmetriccomposition.spatial.centeredcomposition.spatial.offcenter
How can we describe the image in terms of visual texture
composition.texture.naturalcomposition.texture.syntheticcomposition.texture.harmoniouscomposition.texture.dissonantcomposition.texture.smoothcomposition.texture.roughcomposition.texture.crackedcomposition.texture.patterned
Categories and concepts dervied from cinematography and photography concepts
How do we describe the image in terms of camera orientation and placement with respect to height?
shot.angle.na(not applicable)shot.angle.aerialshot.angle.birdseyeshot.angle.highshot.angle.eyelevelshot.angle.low
How do we descrive the image in terms of camera orientation with respect to rotation along the direction of the lens?
shot.level.na(not applicable)shot.level.levelshot.level.tilted
How do we describe the image in terms of cinemographic shot type language?
shot.type.na(not applicable)shot.type.portraitshot.type.twoshotshot.type.mastershot.type.overtheshoulder
How do we describe the image in terms of camera proximity to the subject (typically a person)
shot.framing.na(not applicable)shot.framing.extremecloseupshot.framing.closeupshot.framing.mediumshot.framing.longshot.framing.extremelong
How do we descibe the image in terms of camera focus
shot.focus.na(not applicable)shot.focus.deepshot.focus.shallowshot.focus.out
How do we describe the lighting of the subject in the image
shot.lighting.na(not applicable)shot.lighting.softshot.lighting.hardshot.lighting.lowkeyshot.lighting.highkeyshot.lighting.silhouette
What is the subject of the image, if any?
shot.subject.na(not applicable)shot.subject.animalshot.subject.objectshot.subject.textshot.subject.locationshot.subject.person
If the subject of the image is a person, are we focusing on a particular location on the body?
shot.subject.person.faceshot.subject.person.armshot.subject.person.hand
What is the time of day in the image, if any?
shot.timeofday.na(not applicable)shot.timeofday.twilight (dawn or dusk)shot.timeofday.dayshot.timeofday.night
What is the weather in the image, if any?
shot.weather.na(not applicable)shot.weather.sunnyshot.weather.cloudyshot.weather.rainingshot.weather.snowing
How do we describe the location of the subject / camera in the image, if any? Borrows heavily from ideas in Places 365
shot.location.na(not applicable)shot.location.interior(indoors / inside)shot.location.exterior(outdoors / outside)
Specific nature (exterior) categories if we can identify them
shot.location.exterior.beachshot.location.exterior.cave(cave entrance)shot.location.exterior.desertshot.location.exterior.plainsshot.location.exterior.wetlandsshot.location.exterior.hillsshot.location.exterior.forestshot.location.exterior.mountainshot.location.exterior.polar(artic, antartic)shot.location.exterior.rivershot.location.exterior.lakeshot.location.exterior.oceanshot.location.exterior.canyonshot.location.exterior.glaciershot.location.exterior.spaceshot.location.exterior.sky
Specific township (exterior) categories if we can identify them
shot.location.exterior.cityshot.location.exterior.parkshot.location.exterior.playgroundshot.location.exterior.roadshot.location.exterior.sidewalkshot.location.exterior.suburbshot.location.exterior.town
Specific building (exterior) categories if we can identify them
shot.location.exterior.houseshot.location.exterior.mansionshot.location.exterior.apartmentshot.location.exterior.castleshot.location.exterior.skyscrapershot.location.exterior.palaceshot.location.exterior.officeshot.location.exterior.farmshot.location.exterior.industrialplantshot.location.exterior.restaurantshot.location.exterior.bar(or pub)shot.location.exterior.cafeshot.location.exterior.chruchshot.location.exterior.mosqueshot.location.exterior.synagogueshot.location.exterior.templeshot.location.exterior.cathedralshot.location.exterior.monasteryshot.location.exterior.stadiumshot.location.exterior.theatershot.location.exterior.garageshot.location.exterior.mallshot.location.exterior.portshot.location.exterior.piershot.location.exterior.warehouseshot.location.exterior.ruinsshot.location.exterior.airportshot.location.exterior.station.trainshot.location.exterior.station.gasshot.location.exterior.busstopshot.location.exterior.station.subwayshot.location.exterior.storeshot.location.exterior.hospitalshot.location.exterior.schoolshot.location.exterior.libraryshot.location.exterior.parkinglotshot.location.exterior.bridgeshot.location.exterior.tunnel(entrance)
Specific vehicle (exterior) categories if we can identify them
shot.location.exterior.carshot.location.exterior.busshot.location.exterior.motorcycleshot.location.exterior.bicycleshot.location.exterior.truckshot.location.exterior.trainshot.location.exterior.boatshot.location.exterior.airplaneshot.location.exterior.spacecraft
Specific room (interior) categories if we can identify them
shot.location.interior.caveshot.location.interior.lobbyshot.location.interior.foyershot.location.interior.hallwayshot.location.interior.livingroomshot.location.interior.diningroomshot.location.interior.kitchenshot.location.interior.closetshot.location.interior.bedroomshot.location.interior.bathroomshot.location.interior.closetshot.location.interior.garageshot.location.interior.auditoriumshot.location.interior.gymshot.location.interior.emergencyroomshot.location.interior.studyshot.location.interior.stairwellshot.location.interior.elevatorshot.location.interior.garageshot.location.interior.factory(factory line, factory floor)shot.location.interior.warehouseshot.location.interior.dungeonshot.location.interior.throneroomshot.location.interior.classroomshot.location.interior.cafeteriashot.location.interior.officeshot.location.interior.openofficeshot.location.interior.conferenceroomshot.location.interior.barnshot.location.interior.restaurantshot.location.interior.commercialkitchenshot.location.interior.barshot.location.interior.cafeshot.location.interior.arenashot.location.interior.stageshot.location.interior.dancefloorshot.location.interior.airport(terminal)shot.location.interior.station.train(terminal)shot.location.interior.station.bus(terminal)shot.location.interior.station.subway(subway platform, subway turnstyle, subway car)shot.location.interior.storeshot.location.interior.aisle(store)shot.location.interior.checkout(store)shot.location.interior.mallshot.location.interior.naveshot.location.interior.pulpitshot.location.interior.prayerhallshot.location.interior.synegogueshot.location.interior.meditationshot.location.interior.grandhallshot.location.interior.cryptshot.location.interior.cloister
Specific vehicle (interior) if we can identify them
shot.location.interior.carshot.location.interior.busshot.location.interior.truckshot.location.interior.trainshot.location.interior.subway(subway car)shot.location.interior.boatshot.location.interior.airplane(cockpit, cabin)shot.location.interior.spacecraft(cockpit, cabin)